Mitigating an Attack on the Proton Network

It was recently brought to my attention that a handful of block producer nodes on Proton Mainnet  (including the one that I administer) were "producing empty blocks". An empty block is exactly what it sounds like – a block that contains no data. Apparently, someone was flooding the network with invalid transactions. These transactions were reverting and thus were not being included in the blocks produced by the BP's.

Normally, this isn't particularly problematic, but in this case it was because certain BP's were being overwhelmed by the abnormally high number of invalid transactions, and rather than discarding them and including valid transactions in the block the BP produced, some BP's were simply discarding all transactions and producing blocks with no transactions included at all.

The BP's that were suscpectable to this issue were those that were either running their BP and API nodes on the same server, or, as was in my particular case, seperating the API and producer node by using virtual machines that were hosted on the same bare metal server.

During the last decade that I've been doing Linux administatrion, I've never encountered a situation quite like this. I tend to KVM quite a bit, and it has always been a viable solution for nanaging system resources in their own isolated enviroments. But in this case, the sheer scale of the attack on the XPR network was overwhelming the host system. After all, virtual machines share the system resources of their host.

As you may have assumed, the solution was to spin up another dedicated server and host the block producer on that machine, while migrating the API from KVM and running it straight on the bare metal of the original system. After doing this,  we were no longer producing empty blocks.

If you are configuring a Proton mainnet node, it would be a good move to isolate your producer and API servers physically – they must each run on their own bare metal dedicated servers! Doing this will save you from dealing with unforeseen issues.

Additionally, BP's were instructed to enable some additional settings in their `config.ini`.

disable-subjective-p2p-billing = false
disable-subjective-api-billing = false
subjective-account-decay-time-minutes = 60

That took care of it! No more empty blocks.