Hetzner Falkenstein upcoming downtime and Foundation delegations

On February 3, there was a pre-announced maintenance outage at the Hetzner Helsinki data center. A large portion of the Solana validator nodes housed at this data center were impacted, resulting in a significant amount of delinquent stake and network instability for several hours. The Solana Foundation is working on creating appropriate incentives for validators to naturally disperse across different geographies and ISPs, though much of this effort remains a work in progress.

Hetzner Falkenstein has announced downtime between March 2nd-4th between 3:00 - 5:30am CET, split up into 30min windows for work on various routers. Please refer to their planned maintenance schedule posted here: https://status.hetzner.com/. There is currently ~15% of active network stake located at this facility, all of which is potentially at risk of downtime if preventative action is not taken.

In an effort to prevent a similar destabilizing event during the upcoming outage at Hetzner Falkenstein, the Foundation is reaching out to all validators in Falkenstein to consider temporarily relocating their node during the outage, or to have their Foundation delegation temporarily removed during the outage.

If you are currently running a validator node located in Hetzner Falkenstein, you will receive an email with a form with which you can indicate your choice of two options:

1. You are planning to move your validator to avoid being impacted by the scheduled downtime. Your Foundation stake will not be affected.
2. You will not be moving your node out of Hetzner Falkenstein before the downtime. The Solana Foundation will temporarily remove your Foundation delegation.

If validators choose to be destaked, the Foundation will start the cooldown of their Foundation delegation in the current epoch (epoch 154). This will ensure that their validators will not be included in the leader schedule during the Hetzner downtime during epoch 156.

Note that the Foundation will re-apply delegations after the downtime as a one time courtesy during epoch 156 or 157. It is not the responsibility of the Foundation to monitor data center downtimes for validators in this way going forward.

If validators indicate that they are planning to move their validator off of the Falkenstein data center during the downtime and do not do so, they will be de-staked and ineligible for any future Foundation delegations.

This is intended to be a one-time manual intervention to rebalance the Foundation’s stake delegation. Going forward, we intend to rely on the evolving stake bot program, which will enforce limits on the total amount of stake located in a given data center in order for nodes in that location to remain eligible for a Foundation delegation. As this piece of the stake bot program is not fully tested yet, we are taking this manual action.

6 Likes