Benchmark Performance Testing (aka Individual Time Trials)

Preliminary Results are IN


Hi Validators!

With the impending Serum launch and general mainnet excitement ahead, we’d like to do some benchmark performance testing to see where everyone’s at.

This will help us start to understand hardware configurations, how they impact performance and where we might want to focus investments/grants to help everyone get ‘up to speed’.

For this effort, we need your data! If you’re willing to participate, please fill out the form linked below and upload your singlenode bench-tps results. Please run the benchmark on your Mainnet Beta validator configurations if possible.

I’ll collate and present the final results after we get enough to make it interesting.

Consider this a special ‘stage’ of Tour de SOL. We’re calling it the Individual Time Trial.

Benchmarking your configuration:

  1. Follow Singlenode Testnet benchmark instructions here
  2. Upload performance testing results and hardware configuration information here

Additional Notes:

  • Make sure to stop the mainnet/testnet validator process for the duration of the test, while being mindful to retain < 5% delinquency.
  • If you have GPUs, it would be very informative to get a run both with and without them enabled!
  • The form linked above requires a google account login. We’re not collecting emails but feel free to post your results in this thread if you prefer. Just make sure to include the following info:
    • CPU
    • RAM
    • OS Drive
    • Account Drive
    • GPUs enabled during test? If so, what is your GPU setup?
    • Networks (TdS, Mainnet) that you are running, or plan to run this configuration on.
    • Output of bench-tps
7 Likes

Anyone done this? the bench script has been running really long time 45m, loops as below, not sure if stuck or i need to wait… when I ran this back in Nov it took 3 minutes:

20-08-14T13:49:48.714896719Z WARN  solana_bench_tps::bench] Too many failed transfers... 8047 remaining, 0 verified, 145 failures
[2020-08-14T13:49:48.714902550Z WARN  solana_bench_tps::bench] Too many failed transfers... 8046 remaining, 0 verified, 146 failures
[2020-08-14T13:49:48.715521642Z WARN  solana_bench_tps::bench] Too many failed transfers... 8044 remaining, 0 verified, 148 failures
[2020-08-14T13:49:48.728460662Z INFO  solana_bench_tps::bench] Looping verifications
[2020-08-14T13:49:48.728468557Z INFO  solana_bench_tps::bench] Verifying transfers... 8044 remaining, 0 verified, 148 failures
[2020-08-14T13:49:48.828528247Z INFO  solana_bench_tps::bench]  retrying 1000000000 each to 32768 accounts in 8192 txs
1 Like

It literally finished this minute.

1 Like

Server HP ProLiant DL360 G9 1U 2x Intel 12 Core Xeon E5-2673 V3 2.4 GHz 64GB DDR42133P ECC Reg 2 x SSD 1.92TB Intel Enterprise il_28812

Network TDS. Mainnet
RAM:64Gb

Output:
[2020-08-14T13:37:51.823208827Z INFO solana_bench_tps::bench] Token balance: 1000095000
[2020-08-14T13:37:51.823246412Z INFO solana_bench_tps::bench] Node address | Max TPS | Total Transactions
[2020-08-14T13:37:51.823279624Z INFO solana_bench_tps::bench] ---------------------±--------------±-------------------
[2020-08-14T13:37:51.823298572Z INFO solana_bench_tps::bench] 127.0.0.1:8003 | 81127.99 | 3020276
[2020-08-14T13:37:51.823321024Z INFO solana_bench_tps::bench]
Average max TPS: 81127.99, 0 nodes had 0 TPS
[2020-08-14T13:37:51.823346333Z INFO solana_bench_tps::bench]
Highest TPS: 81127.99 sampling period 1s max transactions: 3020276 clients: 1 drop rate: 0.68
[2020-08-14T13:37:51.823370030Z INFO solana_bench_tps::bench] Average TPS: 33137.88

2 Likes

2 x Hetzner EX62-NVME
TdS, Mainnet
Average max TPS: 70100.88, 0 nodes had 0 TPS
Highest TPS: 70100.88 sampling period 1s max transactions: 2546564 clients: 1 drop rate: 0.70
Average TPS: 27846.11

2 Likes

you have to tell it when to stop, like this:
NDEBUG=1 ./multinode-demo/bench-tps.sh --duration 90

1 Like

IBM Cloud Bare Metal

RAM 2x 16GB Micron 16GB DDR4 2Rx4
Processor 2x 2.1GHz Intel Xeon-Skylake (4110-SILVER)
Power Supply 2x SuperMicro PWS-1K02A-1R
Network Card SuperMicro AOC-2UR66-i4XTF
Motherboard SuperMicro X11DPU+_R1.10
GPU NVIDIA TESLA K80

[2020-08-16T02:40:11.985094846Z INFO solana_bench_tps::bench] Node address | Max TPS | Total Transactions [2020-08-16T02:40:11.985104288Z INFO solana_bench_tps::bench] ---------------------+---------------+-------------------- [2020-08-16T02:40:11.985112954Z INFO solana_bench_tps::bench] 127.0.0.1:8003 | 69403.67 | 3158768 [2020-08-16T02:40:11.985125004Z INFO solana_bench_tps::bench] Average max TPS: 69403.67, 0 nodes had 0 TPS [2020-08-16T02:40:11.985134635Z INFO solana_bench_tps::bench] Highest TPS: 69403.67 sampling period 1s max transactions: 3158768 clients: 1 drop rate: 0.62 [2020-08-16T02:40:11.985145821Z INFO solana_bench_tps::bench] Average TPS: 34653.04

1 Like

why 90? I didn’t see that in the instructions, is that just a number you made up?

1 Like

Ryzen 3700X,16GB RAM, 1TB SSD Nvme, 1x NVIDIA 2080TI

Only CPU:

Average max TPS: 43961.43, 0 nodes had 0 TPS
Highest TPS: 43961.43 sampling period 1s max transactions: 3392569 clients: 1 drop rate: 0.76
Average TPS: 37507.06

With GPU

Average max TPS: 114793.67, 0 nodes had 0 TPS
Highest TPS: 114793.67 sampling period 1s max transactions: 7336420 clients: 1 drop rate: 0.57
Average TPS: 80683.016

1 Like

VMware running on 1x Intel® Core™ i7-4770 CPU @ 3.40GHz
Virtual machine has 24GB of RAM. 8 cores.
VM is running on Raid 5 10GB NFS Shared Storage using 3x Intel SSD S3700
NVIDIA T4 GPU Enabled
Local Network

bench-tps output:

Sampler 76259.93 TPS, Transactions: 76305, Total transactions: 9468865 over 301 s
Max TPS | Total Transactions
76259.93 | 9468865

Average TPS: 31374.154

1 Like

completed & submitted results via form

1 Like

2x Intel Xeon E5-2620v4 @ 2.1Ghz (8 core/16 HT, 20MB cache each CPU)
64 GB RAM (8x 8GB DD4 2133)
4x 300GB 10K SAS HDD in RAID 5
OS & Account drive separate LVM LVs in same PV
No GPUs
TdS

With default trace level logging on validator
[2020-08-17T18:55:16.858430309Z INFO solana_bench_tps::bench] Average max TPS: 95511.38, 0 nodes had 0 TPS
[2020-08-17T18:55:16.858445036Z INFO solana_bench_tps::bench] Highest TPS: 95511.38 sampling period 1s max transactions: 4197121 clients: 1 drop rate: 0.60
[2020-08-17T18:55:16.858465790Z INFO solana_bench_tps::bench] Average TPS: 46093.523

With RUST_LOG=warn on validator
[2020-08-17T19:12:00.660520328Z INFO solana_bench_tps::bench] Average max TPS: 103514.71, 0 nodes had 0 TPS
[2020-08-17T19:12:00.660528112Z INFO solana_bench_tps::bench] Highest TPS: 103514.71 sampling period 1s max transactions: 4583238 clients: 1 drop rate: 0.55
[2020-08-17T19:12:00.660540872Z INFO solana_bench_tps::bench] Average TPS: 50689.96

1 Like

@eric what’s the timeline for this? I have some server reshuffling to do and would like to participate.

1 Like

End of week should be fine!

2 Likes

3960X + 64G + 1x2080Ti
Highest TPS: 185089.05
Average TPS: 155914.94

1 Like

Submitted via google form just now.

Anson @ Forbole

2 Likes

Did any of you experience issues enabling CUBA for the benchmarking script? There were some hiccup for me using CUDA-11

1 Like

Hi mate, I believe it wants Cuda 10.1 update 1

As you mentioned, the most recent one is 11, but you can find the older version on nvidia’s website.

1 Like

I think CUDA 11 is not supported by some libraries that are used in Solana.

1 Like

Thanks G1715 and Godonyat, can you share any reference on this topic if you have it handy? I was able to get it running at the end, but 4 x 2080 Ti display cards only can get my average TPS from 38K to 44K… I am quite disappointed about that.