Updates to Foundation Delegation Strategy

The Solana Foundation is implementing two updates to its auto-delegation strategy. First, we’ve implemented a change to the skip-rate threshold calculation (see Block Production Skip Rate below). Second, an infrastructure concentration threshold has been put in place (Infrastructure Concentration below) to try to reduce validator and stake concentration at individual data centers.

These changes will go into effect as of 3 March, however note that the infrastructure concentration feature will be rolled out in phases, as described below.

Block Production Skip Rate

Up until now, there has been a static threshold set on a validator’s block production skip rate, in order to receive a Foundation delegation. A validator’s block production is defined as the ratio of the number of blocks that a validator produces that are confirmed by the network to the number of slots that same validator was scheduled to be the leader/block producer. Skip rate is [1 - block production], so a validator that produced confirmed blocks in 90% of its scheduled leader slots has a skip rate of 10%. A validator whose skip rate exceeded the static threshold would have its Foundation delegation deactivated until the validator’s skip rate dropped back below the threshold.

This threshold calculation has now been updated in the following way:

  1. At the end of each epoch, the unweighted average of all block producer skip rates is calculated.
  2. A fixed threshold for maximum skip rate is added to the cluster average.
  3. Validators who have skip rates above the new threshold have their delegations removed for the subsequent epoch

As an example if, for epoch N, the average validator skip rate was 22%. The auto-delegation bot will add a flat 15% skip rate threshold to the average skip rate (i.e. 22%), resulting in a final skip rate threshold of 37%. So any validators who had a skip rate > 37% in epoch N, will have their Foundation delegations removed as of epoch N+1, until their skip rate falls below the threshold set by this algorithm in future epochs.

Infrastructure Concentration

In an effort to encourage a reduction in validator infrastructure concentration, the foundation will be adding a data center location component to its staking strategy. Specifically, the Foundation auto-delegation bot will calculate the total amount of stake concentrated under all identifiable unique data center locations, if the stake associated with any single data center is above a predetermined threshold (e.g. 25%) all of the Foundation delegations to validators operating within that data center will be removed, until the stake falls below the threshold.

Data will come from the validators.app API and be sampled at stake bot run time, like all other criteria. Since this new obligation requires non-trivial action be taken by affected operators, its enforcement will be phased in as follows.

Phase 1: March 3rd - 17th

Starting on 3 March, the Mainnet-Beta stake-o-matic will begin issuing warnings to validators hosted in data centers with a combined stake over 25% that they are affected. These warnings will come in the form of messages in the #mb-stake Discord channel. Note that during this first phase only warning messages will be issued, NO STAKE WILL BE REMOVED.

Phase 2: March 17th - April 17th

On 17 March, validators which were affected by the Feb. 3, 2021 Hetzner Helsinki planned maintenance network outage will become eligible for destaking. A list of validators IDs that were impacted by the Hetzner incident, and therefore eligible for destaking via this infrastructure concentration threshold, will be published prior to this phase.

This initial limited phase is intended as both a trial of the destaking feature and as an opportunity for those operators affected by the outage to prove that they have taken appropriate measures in the wake of the mass outage incident.

Phase 3: April 17th+

Finally, all validators will become eligible for infrastructure concentration based destaking on 17 April. The threshold concentration will be determined during the preceding week, based on that of the worst case facility and announced in advance. The threshold concentration will then be reduced by 5% every 30 days until the target maximum concentration of 15% per data center is reached.

3 Likes

@eric Q1: will bot count only foundation delegated stake or all stake at the location? for example if P2P moves to hetzner (temporarily for maintenance for example) - will it affect those who are hosting there?

or for example some entity stakes 100M sol (just example) to some random validator at hetzner - will it affect those who are at hetzner?

4 Likes

Q2: will bot count all stake under ASN as one single DC or will it count separate datacenters? for example OVH has about 10+ locations all over the globe (7 in europe/canada, 2 US + 2 Asia) but all of them share the same ASN 16276 (not sure 100% if US/Asia also in this ASN)

also notice that currently validators.app has problems correctly identifying separate datacenters: almost all OVH is placed under “Paris” while being separate DCs - Gravelines, Roubaix, London, Frankfurt and so on

I perfectly understand that geo-location by IP is and will be flawed by design and sometimes identifying one physical DC from another is impossible. just asking to clarify the rules

5 Likes

Q1

It will count all stake at a location. So, it is possible that a node with a large amount of stake could impact other smaller staked nodes at a given location. Not ideal, but this is all a work in progress. Features such as stake throttling and stake pools will hopefully minimize the impact capable from a single, large staked, node.

Q2

On first launch, the bot will count all stake under a given DC (not under a given ASN), as labelled via validators.app. The hope is that we get to a place where we are setting thresholds at the ASN level though.

@brian.long has been helpful clarifying validators.app resolution :slight_smile:

1 Like

Hi, the commercial IP geo-locator services do a poor job with OVH IP addresses. I’ll write a special script to grab locations through traceroute. For now, OVH is well below 25%, so this is really a matter of making things “look right.” Cheers

1 Like

@eric as Brian says, the OVH info isn’t accurate at the moment. If you look at OVH Paris grouping, for example, it shows many validators that are in other OVH data centers in other geo locations.

It will count all stake at a location. So, it is possible that a node with a large amount of stake could impact other smaller staked nodes at a given location.

looks understandable but a bit weird. Some thoughts on this:

  • communicate foundations’ will to large nodes be outside of popular places like hetzner/OVH/ikoula even for their secondary nodes. any node with stake above 1M can easily have any provider paid and smaller players should not suffer from big guys movements that they (small players) are not capable of predict. i imagine that it might be very disappointing and demotivating if you loose delegation for the thing you did not do
  • something like LIFO - the newer node at tds gets more chance to loose stake in an event of datacenter overstake than an older node.
2 Likes

@eric please give any comment on my last thoughts. is it possible/acceptable ?

currently there are about 25% stake at helsinki. I’m not there but i see many Russian node operators questioning the situation - will P2P or Melea come there for maintenance (they did some time ago)? Or will Everstake/Forbole (about 7% combined) leave? Will older guys (who started from SLP) be safe or everybody will be destaked when additional 2% comes to Helsinki (not hard to imagine imagine that Forbole/Everstake attracts new big client)?

communicate foundations’ will to large nodes be outside of popular places like hetzner/OVH/ikoula even for their secondary nodes

:100: Larger nodes should be in more of a position to not have to rely on Hetzner

Yes, we’re considering de-staking algorithms to be implemented in a subsequent version. I’m not sure if LIFO is the right way to approach this but the issue is on our radar!

Also, with community run stake pools, larger nodes that act like bullies like this could get ‘punished’ by the general community and ‘de-staked’ from stake pools

Hello! Where we can see a list of validators IDs that were impacted by the Hetzner incident according Phase 2: March 17th - April 17th?

Hi @eric .
Taking into account penalties for new test validators in Hetzner, does it means that Solana prefers validators from small Data center?

One more question: if validator is apporved for mainnet - is it ok to have bothh nodes in one DC?

Thank you
Ivan