RESULTS - Individual Time Trial / Benchmark Performance Testing

Validators 🥩
#1

Hi Validators!

Thanks to everyone who submitted their performance data in Tour de SOL’s first Individual Time Trial. I’ve presented some of the high-level results below. I think this data will be useful for sometime as we become more familiar with it and are able to use it as a resource for performance and hardware related questions.

Note: the results below may not be 100% complete. The data collection wasn’t super rigorous so some of the submissions didn’t have enough information to be reliable. Also, for recent and future submissions, it may be more practical to compare your results to the results below on your own, rather than me trying to keep everything updated.

To date, we’ve collected over 70 benchmark tests along with hardware config info for those tests. I’ve decided to not post identifying information linked to the performance results for the following reasons

  1. I didn’t say I would in advance
  2. Many people submitted performance results of sub-optimal configurations or hardware that they were curious about. Also, many folks submitted multiple benchmarks for slightly different configurations. For these reasons these results don’t necessarily reflect real-life validator performance on Mainnet or Tour de SOL. Posting validator names/ID along with the results might be misleading to the performance capabilities of any given validator.

Results

Each row in the table below corresponds to one performance submission, not aggregated results of a particular hardware configuration.

Leaderboard: Avg TPS

cpu gpu tps_avg
:trophy: AMD Ryzen 3950x NVIDIA RTX 3080 250862.97
2x Intel Xeon Gold 6230 6x 2080ti 241053
AMD Threadripper 3970x 201438.88
Ryzen TR 3960X 1 x Nvidia 2080 Ti 155914.94
2x Intel Xeon Gold 6230 137131
AMD EPYC™ 7502P 32-Core “Rome” 136086.00
AMD EPYC™ 7502P 32-Core 132600.00
AMD Ryzen Threadripper 2950X 16-Core Processor 2x NVIDIA-SMI 418.67 Driver Version: 418.67 CUDA Version: 10.1 95193.03
AMD Ryzen 3900X 1x 2080Ti. 92637.63
AMD EPYC 7502P 32 Cores 64 Threads 87302.53
Epyc 7502p 84584.00
Ryzen 3700X 1x Nvidia 2080TI 80683.00
AMD Threadripper 2950x 3 x Nvidia 2080 Ti 79205.18
2 x Intel Xeon E5 2630v4, (2 x 10 x 2.20 GHz), 40 Threads in total, 2 physical Intel CPUs, each with 10 core 79175.95
1x AMD EPYC 7282, 16 x 2.8 GHz, 32 Threads in total 77200.00
AMD EPYC 7282, 16 x 2.8 GHz, 32 Threads in total 76209.96
1x AMD EPYC 7282, 16 x 2.8 GHz, 32 Threads in total 76209.00
AMD EPYC 7282 75738.33
AMD EPYC 7502P, 32 Core “Rome” 75382.90
AMD Ryzen Threadripper 2950X 16-Core Processor 73434.04
Intel® Xeon® Gold 6140 CPU @ 2.30GHz N/A 73130.89
AMD EPYC 7282 72049.57
1x AMD EPYC 7282, 16 x 2.8 GHz, 32 Threads in total 72047.00
2x AMD EPYC 7282, 16 x 2.8 GHz, 32 Threads in total 72028.00
1x AMD EPYC 7282, 16 x 2.8 GHz, 32 Threads in total 71857.00
2x AMD EPYC 7282, 16 x 2.8 GHz, 32 Threads in total 69812.00
2x AMD EPYC 7282, 16 x 2.8 GHz, 32 Threads in total 69410.00
2x AMD EPYC 7282, 16 x 2.8 GHz, 32 Threads in total 68376.00
2x AMD EPYC 7282, 16 x 2.8 GHz, 32 Threads in total 67118.00
AMD3900X 65035.09
AMD EPYC 7282 16-Core Processor NA 64792.11
AMD Ryzen 9 3900 12-Core 64714.17
Intel Xeon E5-2697 v3 58178.00
(32) Intel Xeon Gold 6140 @2.3GHz 57385.84
AMD EPYC 7401P 24-Core 54964.35
2 x Intel Xeon E5 2630v4 53673.00
2 x Intel Xeon E5 2630v4, (2 x 10 x 2.20 GHz), 40 Threads in total, 2 physical Intel CPUs, each with 10 core 53456.99
2 x Intel Xeon E5 2630v4 53188.00
2 x Intel Xeon E5 2630v4, each CPU with 10 cores x 2.20 GHz, 40 Threads totally 53169.54
AMD EPYC 7401P 52885.4
Ryzen 3900X 1 x nVidia Quadro RTX 4000 52829.66
2 x Intel Xeon E5 2630v4 52048.00
Intel Xeon E5 2630v4 51958.33
2 x Intel Xeon E5 2630v4 51816.00
Intel® Xeon® Silver 4108 CPU @ 1.80GHz 4x Nvidia 2080Ti 51275.60
2 x Intel Xeon E5 2630v4 50931.00
2 x Intel Xeon E5 2630v4, (2 x 10 x 2.20 GHz), 40 Threads in total, 2 physical Intel CPUs, each with 10 core 50435.05
AMD EPYC 7502P 50276.60
2 x Intel Xeon E5 2630v4, (2 x 10 x 2.20 GHz), 40 Threads in total, 2 physical Intel CPUs, each with 10 core 49920.09
Intel Xeon E5 2630v4 47603.01
AMD EPYC 7502P 32 Core 46509.25
Intel vCPU (31 utilized) 46498.86
AMD Threadripper 2950x 45077.98
2 x Intel® Xeon® CPU E5-2680 v4 @ 2.40GHz 4 x TU102 [GeForce RTX 2080 Ti Rev. A] 44773.33
2 x Intel® Xeon® CPU E5-2680 v4 @ 2.40GHz 4 x TU102 [GeForce RTX 2080 Ti Rev. A] 44408.11
2x Intel 12 Core Xeon E5-2673 V3 2.4 GHz GPU 2x1080TI 44362.88
Intel® Core™ i7-6700 1x NVIDIA GTX 1080 42388.50
TDS: AMD Ryzen Threadripper 2950X 16-Core Processor 42015.00
AMD EPYC 7282 16 x 2.80 GHz 41149.73
AMD Ryzen 7 3700X Octa-Core 40004.47
AMD EPYC 7282 2.8 GHz 8 core 38714.56
AMD Ryzen 7 3700X 38499.37
2 x Intel® Xeon® CPU E5-2680 v4 @ 2.40GHz 38068.84
AMD Ryzen 7 3700X Octa-Core 37608.81
Ryzen 3700X 37507.00
AMD Ryzen 7 3700X Octa-Core 36322.32
amd epyc 7402P - 35548.00
16VCPU’s of AMD EPYC 35232.02
2x 2.1GHz Intel Xeon-Skylake (4110-SILVER) - 32 core total Tesla K80 34653.04
16 CPU, amd epyc gen 2 34406.25
2x Intel 12 Core Xeon E5-2673 2.4 GHz 34215.88
Intel® Core™ i9-9900K Octa-Core incl. Hyper-Threading Technology 33480.00
2 x 12 Xeon E5-2673 V3 2.4 GHz 33137.88
8 core AMD EPYC 7282 2.8 GHz 31453.27
Intel® Core™ i9-9900K Octa-Core 29745.37
2 Intel Xeon E5 2630v4 (10 x 2.20 GHz) 29621.59
VMware running on 1x Intel® Core™ i7-4770 CPU @ 3.40GHz 1x Nvidia T4 28608.72
Intel® Core™ i9-9900K Octa-Core 28311.67
Intel® Core™ i9-9900K Octa-Core incl. Hyper-Threading Technology 28271.00
AMD Ryzen 7 3700X Octa-Core 28242.75
Intel® Core™ i9-9900K 27846.11
AMD Ryzen 5 3600 Hexa-Core 26617.11
12vCPUs 25230.22
Intel® Core™ i7-8700 Hexa-Core Coffee Lake 23476.15
Intel® Core™ i9-9900K CPU @ 3.60GHz 22805.00
Intel® Core™ i9-9900K CPU @ 3.60GHz 22079.00
AMD EPYC 7000 32vCPU 19114.47
Ryzen 5 3600 18935.11
AMD Ryzen 5 3600 4.2GHz 18935.11
Intel® Core™ i7-6700 17098.73
AMD EPYC 7282 16-Core Processor 13796.25
VMware running on 1x Intel® Core™ i7-4770 CPU @ 3.40GHz 12667.73
Intel® Core™ i7-7700 CPU @ 3.60GHz 12140.30

Results: Max TPS

cpu gpu tps_max
:trophy: AMD Ryzen 3950x NVIDIA RTX 3080 430698.47
2x Intel Xeon Gold 6230 6x 2080ti 364659
AMD Threadripper 3970x 271025.56
2x Intel Xeon Gold 6230 244303
AMD EPYC™ 7502P 32-Core “Rome” 228234.00
AMD EPYC™ 7502P 32-Core 202545.00
Ryzen TR 3960X 1 x Nvidia 2080 Ti 185089.05
1x AMD EPYC 7282, 16 x 2.8 GHz, 32 Threads in total 170245.00
Intel® Xeon® Gold 6140 CPU @ 2.30GHz N/A 146496.45
AMD Ryzen 3900X 1x 2080Ti. 139387.11
Intel® Xeon® Silver 4108 CPU @ 1.80GHz 4x Nvidia 2080Ti 134303.30
AMD3900X 125644.49
2 x Intel® Xeon® CPU E5-2680 v4 @ 2.40GHz 125604.41
AMD Ryzen Threadripper 2950X 16-Core Processor 124909.04
AMD Ryzen 9 3900 12-Core 123015.72
Intel Xeon E5-2697 v3 121190.00
Epyc 7502p 115211.00
AMD EPYC 7282 114948.24
Ryzen 3700X 1x Nvidia 2080TI 114793.00
2x AMD EPYC 7282, 16 x 2.8 GHz, 32 Threads in total 114545.00
AMD Ryzen Threadripper 2950X 16-Core Processor 2x NVIDIA-SMI 418.67 Driver Version: 418.67 CUDA Version: 10.1 113988.84
2x AMD EPYC 7282, 16 x 2.8 GHz, 32 Threads in total 111823.00
AMD Threadripper 2950x 3 x Nvidia 2080 Ti 111580.80
2 x Intel® Xeon® CPU E5-2680 v4 @ 2.40GHz 4 x TU102 [GeForce RTX 2080 Ti Rev. A] 110876.01
2x AMD EPYC 7282, 16 x 2.8 GHz, 32 Threads in total 109660.00
AMD EPYC 7282 107938.42
1x AMD EPYC 7282, 16 x 2.8 GHz, 32 Threads in total 106739.00
AMD EPYC 7282, 16 x 2.8 GHz, 32 Threads in total 106088.48
1x AMD EPYC 7282, 16 x 2.8 GHz, 32 Threads in total 106088.00
2x AMD EPYC 7282, 16 x 2.8 GHz, 32 Threads in total 105938.00
2x AMD EPYC 7282, 16 x 2.8 GHz, 32 Threads in total 105052.00
2 x Intel Xeon E5 2630v4 104567.00
1x AMD EPYC 7282, 16 x 2.8 GHz, 32 Threads in total 103404.00
AMD EPYC 7502P 32 Cores 64 Threads 102168.00
(32) Intel Xeon Gold 6140 @2.3GHz 99688.98
2 x Intel Xeon E5 2630v4 97682.00
2x Intel 12 Core Xeon E5-2673 V3 2.4 GHz GPU 2x1080TI 96755.99
Ryzen 3900X 1 x nVidia Quadro RTX 4000 94935.80
2 x Intel Xeon E5 2630v4 94905.00
AMD EPYC 7282 16-Core Processor NA 94799.77
AMD EPYC 7502P, 32 Core “Rome” 94421.27
2 x Intel Xeon E5 2630v4 93968.00
AMD EPYC 7502P 92284.55
2 x Intel Xeon E5 2630v4 90958.00
2 x Intel® Xeon® CPU E5-2680 v4 @ 2.40GHz 4 x TU102 [GeForce RTX 2080 Ti Rev. A] 88217.34
2 x Intel Xeon E5 2630v4, (2 x 10 x 2.20 GHz), 40 Threads in total, 2 physical Intel CPUs, each with 10 core 86731.28
2x Intel 12 Core Xeon E5-2673 2.4 GHz 85327.53
Intel Xeon E5 2630v4 82482.95
2 x 12 Xeon E5-2673 V3 2.4 GHz 81127.99
Intel Xeon E5 2630v4 79401.12
AMD EPYC 7401P 24-Core 76781.00
2 x Intel Xeon E5 2630v4, each CPU with 10 cores x 2.20 GHz, 40 Threads totally 76510.93
AMD EPYC 7401P 74717.75
2 x Intel Xeon E5 2630v4, (2 x 10 x 2.20 GHz), 40 Threads in total, 2 physical Intel CPUs, each with 10 core 73836.75
8 core AMD EPYC 7282 2.8 GHz 73518.95
2 x Intel Xeon E5 2630v4, (2 x 10 x 2.20 GHz), 40 Threads in total, 2 physical Intel CPUs, each with 10 core 72524.35
AMD EPYC 7502P 32 Core 72350.04
AMD Ryzen 7 3700X 71583.65
Intel® Core™ i9-9900K 70100.88
2x 2.1GHz Intel Xeon-Skylake (4110-SILVER) - 32 core total Tesla K80 69403.67
Intel vCPU (31 utilized) 69313.61
Intel® Core™ i9-9900K Octa-Core 69135.86
Intel® Core™ i9-9900K Octa-Core 69010.24
amd epyc 7402P - 66746.00
AMD Threadripper 2950x 64607.55
Intel® Core™ i9-9900K Octa-Core incl. Hyper-Threading Technology 63839.00
TDS: AMD Ryzen Threadripper 2950X 16-Core Processor 61934.00
Ryzen 5 3600 59976.04
AMD Ryzen 5 3600 4.2GHz 59976.04
Intel® Core™ i7-8700 Hexa-Core Coffee Lake 58523.17
AMD Ryzen 7 3700X Octa-Core 58520.85
AMD Ryzen 7 3700X Octa-Core 58245.02
Intel® Core™ i7-6700 1x NVIDIA GTX 1080 57598.21
AMD EPYC 7000 32vCPU 54053.70
AMD Ryzen 7 3700X Octa-Core 53428.13
AMD EPYC 7282 2.8 GHz 8 core 52015.38
AMD Ryzen 7 3700X Octa-Core 51151.96
AMD EPYC 7282 16 x 2.80 GHz 51064.62
VMware running on 1x Intel® Core™ i7-4770 CPU @ 3.40GHz 1x Nvidia T4 48610.36
16VCPU’s of AMD EPYC 47962.34
2 x Intel Xeon E5 2630v4, (2 x 10 x 2.20 GHz), 40 Threads in total, 2 physical Intel CPUs, each with 10 core 47002.96
16 CPU, amd epyc gen 2 45462.57
12vCPUs 44431.22
Ryzen 3700X 43961.00
AMD Ryzen 5 3600 Hexa-Core 42652.69
2 Intel Xeon E5 2630v4 (10 x 2.20 GHz) 41002.35
Intel® Core™ i9-9900K CPU @ 3.60GHz 40916.00
Intel® Core™ i9-9900K Octa-Core incl. Hyper-Threading Technology 40275.00
Intel® Core™ i9-9900K CPU @ 3.60GHz 34422.00
Intel® Core™ i7-6700 34158.39
VMware running on 1x Intel® Core™ i7-4770 CPU @ 3.40GHz 27053.77
AMD EPYC 7282 16-Core Processor 22436.41
Intel® Core™ i7-7700 CPU @ 3.60GHz 20099.04

If we step back and look at the distributions, we see a fairly wide range of performance results.

h_avg_max_tps
h_avg_max_tps3600Ă—2025 217 KB

It’s hard to see, but the addition of GPU(s) does seem to improve the performance of avg TPS (this is confirmed in the regression analysis below):

h_avg_tps_gpu_enabled
h_avg_tps_gpu_enabled3600Ă—2025 219 KB

While it also seems to slightly improve max TPS (we also see this in the regression results below):

h_max_tps_gpu_enabled
h_max_tps_gpu_enabled3600Ă—2025 214 KB

Simply segmenting by whether a test was run with GPU(s) or not, shows the average improvement to max tps:

gpu_enabled avg_max_tps avg_avg_tps
No 85298.6 51394.19
Yes 105041.5 65148.74

I.e., very roughly, GPUs tend to add ~20-25% increase to tps performance. See the multivariate analysis below to see this effect after controlling for some basic CPU parameters.

Regressions

If we look at CPU contributions to avg and max TPS, simple regressions suggest that processor speed and GPU(s) likely have significant impact on both avg and max TPS.

tps_avg ~ n_cores + n_threads + processor_speed_ghz + gpu_enabled:

Estimate Std. Error t value Pr(>|t|)
(Intercept) -50750.155413 21033.349 -2.4128424 0.0185716
n_cores 2969.839862 2057.681 1.4432949 0.1535969
n_threads 8.494533 1031.444 0.0082356 0.9934535
processor_speed_ghz 17947.264097 5744.019 3.1245133 0.0026317**
gpu_enabledYes 20866.339528 7991.642 2.6110204 0.0111292**

tps_max ~ n_cores + n_threads + processor_speed_ghz + gpu_enabled:

Estimate Std. Error t value Pr(>|t|)
(Intercept) -18200.59792 31588.111 -0.5761851 0.5664207
n_cores 3466.27688 3090.247 1.1216828 0.2660013
n_threads 14.35673 1549.034 0.0092682 0.9926327
processor_speed_ghz 15236.48295 8626.430 1.7662559 0.0819082*
gpu_enabledYes 29743.38676 12001.934 2.4782162 0.0157300*

These results suggest that each additional GHz of processor speed might increase avg performance by ~20 kTPS and max performance ~15 kTPS.

They also build on the GPU results from above, suggesting that after controlling for CPU factors, the addition of GPU(s) has a significant impact on performance. I’ve held off trying to read tea-leaves from GPU sub-analysis just because we don’t have that much data. As this dataset grows this will be something to keep an eye on.

One aspect that I have neglected to take into consideration, and that I’d like to eventually get around to investigating, is CPU/GPU generation. I split out N cores, N threads and processor speed, but haven’t coded generation or manufacturer. So, e.g., a Threadripper 2950x will be indistinguishable from an EPYC 7282 (or 2x Xeon E5-2680) in these analyses. See the anonymized dataset below if anyone feels motivated to tackle this.

Solana’s first ITT has been fun and has provided some interesting data, I hope folks found it useful. You can find an anonymized version of the raw data here.

Update @pkrasam put together a dockerized version of the ITT scripts for easy deployment here: https://github.com/pkrasam/solana-itt

4 Likes
Benchmark Performance Testing (aka Individual Time Trials)
#2

Big mama - STRx AMD 3970

#3

Do we have RAM or SSD speeds?

#4

Hi Eric, thank you for providing some considered statistical analysis of how TPS performance varies with hardware configuration. As you say we need to get more data in order to extract more information. I presume you will therefore want to continue to gather this information; probably anonymized given your comments.

#5

Yeah, the original ITT thread will stay open and we’ll continually push folks to submit through the performance submission form.
From this (and future) analysis, I’ll likely continually adjust the form to get better data as well

#6

We have RAM numbers but SSD speeds would be a pain. The data is there (see link at end of post) if anyone wants to go entry by entry looking up the speeds for the specific drives.
I’ll try to throw some of the RAM info up, but at first glance it only had a weak impact on performance.