This topic discusses the issue with the low amount of transactions in the leader blocks of the Solana validator we faced in P2P validator after upgrading from v1.8.14 to v1.9.12.
Once we upgraded our validator to v1.9 we found that transaction rewards reduced dramatically. Since the validator earns rewards from transactions included to the block, we checked the average amount of transactions in our leader blocks. As you can expect we found abnormal values of around 500 transactions per block which is not normal when your validator produces >90% of blocks like that. Nowadays, the average number of transactions per healthy block is 1.4k-1.6k give or take.
As a curious moment, we didn’t detect any problem with skip-rate of leader blocks. In fact, skip-rate was one of the lowest within super minority validators whereas the number of transactions were fairly low.
Let’s talk about why the amount of transactions in a block is important? There are the main points below:
- If a validator generates only “small blocks” (with mostly vote transactions) then it means end-users wait more time until their transactions are included in the blockchain. So, it is about the user experience of using Solana (which is highly important)
- A validator doesn’t earn a transaction fee as much as it can.
At this stage of discussion, it would be great to see a full picture of how the average transaction per block depends on the Solana version for all validators. We are especially interested in looking at 287-290 epochs when the majority of validators updated to v1.9. Let’s look at the Chart 1:
Chart 1. The average amount of transactions in the block for all validators in mainnet-beta
As you can see on Chart 1, updating to v1.9 affected cluster performance in some way.
In the red square, we see a significant bunch of validators whose average transactions per block dropped by 25%.
In the green square - a small number of validators whose performance drastically dropped.
In this stage of troubleshooting, we understood that some of the tuning applied to our Solana validator setup interfered with Solana v1.9. After some debugging, we found that the variable
SOLANA_BANKING_THREADS was the issue in our case. Once the value was lowered to the default level, the problem was gone for our validator.
A few words about
SOLANA_BANKING_THREADS. It limits the number of transaction processing threads during leader slots of the validators and is set up via an environment variable for the solana-validator process. The default value is 4. It looks like some updates in a transaction processing pipeline landed in the branch v1.9 was a cause of behavior we faced.
After the troubleshooting, we got back to normal (1.4k-1.6k transactions per block) and brought transaction rewards back to a normal level.
We have prepared a dashboard where maintainers of Solana validators can check how validators performed regarding the amount of transactions per block in a timeline between 287 and 293 epochs. The screenshot of this dashboard is presented in Chart 2. The data used on this dashboard is not live and covers only a fixed period (286-293 epoch) when solana mainnet-beta cluster updated from 1.8 to 1.9 version. The URL of the dashboard is here.
Chart 2. Public dashboard with average transaction per block for all validators
We hope that all of the above will be helpful to every Solana validator and help you make profitable blocks.