Issue with low amount of transactions in the leader blocks

Overview

This topic discusses the issue with the low amount of transactions in the leader blocks of the Solana validator we faced in P2P validator after upgrading from v1.8.14 to v1.9.12.

Needs

Once we upgraded our validator to v1.9 we found that transaction rewards reduced dramatically. Since the validator earns rewards from transactions included to the block, we checked the average amount of transactions in our leader blocks. As you can expect we found abnormal values of around 500 transactions per block which is not normal when your validator produces >90% of blocks like that. Nowadays, the average number of transactions per healthy block is 1.4k-1.6k give or take.

As a curious moment, we didn’t detect any problem with skip-rate of leader blocks. In fact, skip-rate was one of the lowest within super minority validators whereas the number of transactions were fairly low.

Let’s talk about why the amount of transactions in a block is important? There are the main points below:

  1. If a validator generates only “small blocks” (with mostly vote transactions) then it means end-users wait more time until their transactions are included in the blockchain. So, it is about the user experience of using Solana (which is highly important)
  2. A validator doesn’t earn a transaction fee as much as it can.

Solution

At this stage of discussion, it would be great to see a full picture of how the average transaction per block depends on the Solana version for all validators. We are especially interested in looking at 287-290 epochs when the majority of validators updated to v1.9. Let’s look at the Chart 1:


Chart 1. The average amount of transactions in the block for all validators in mainnet-beta

As you can see on Chart 1, updating to v1.9 affected cluster performance in some way.

In the red square, we see a significant bunch of validators whose average transactions per block dropped by 25%.

In the green square - a small number of validators whose performance drastically dropped.

In this stage of troubleshooting, we understood that some of the tuning applied to our Solana validator setup interfered with Solana v1.9. After some debugging, we found that the variable SOLANA_BANKING_THREADS was the issue in our case. Once the value was lowered to the default level, the problem was gone for our validator.

A few words about SOLANA_BANKING_THREADS. It limits the number of transaction processing threads during leader slots of the validators and is set up via an environment variable for the solana-validator process. The default value is 4. It looks like some updates in a transaction processing pipeline landed in the branch v1.9 was a cause of behavior we faced.

Conclusions

After the troubleshooting, we got back to normal (1.4k-1.6k transactions per block) and brought transaction rewards back to a normal level.

We have prepared a dashboard where maintainers of Solana validators can check how validators performed regarding the amount of transactions per block in a timeline between 287 and 293 epochs. The screenshot of this dashboard is presented in Chart 2. The data used on this dashboard is not live and covers only a fixed period (286-293 epoch) when solana mainnet-beta cluster updated from 1.8 to 1.9 version. The URL of the dashboard is here.


Chart 2. Public dashboard with average transaction per block for all validators

We hope that all of the above will be helpful to every Solana validator and help you make profitable blocks.

1 Like

very cool analysis, perhaps better to post this on github as an issue where the core devs can review this?

Thanks for the feedback! Based on the data the problem is not massive. But the observation is important. We have shared the links in discord and created are issue so that core devs are aware of it.

Great breakdown as always @pvlpvlv ! :smiley: