Rollback After a Validator Upgrade Leaves the Database Unopenable

Besu migrates its database schema on first start of a newer release, and those migrations are forward-only. Once the migration has run, the on-disk format carries a version marker that the older binary does not understand, so the process exits rather than risk reading a layout incorrectly. The binary downgrade is not a supported operation on a migrated data directory.

The real options:

1. Restore the pre-upgrade snapshot. Four days old, so v2 will need to catch up a manageable number of blocks. The restored snapshot must be paired with the older release - this is the fastest realistic path and the reason snapshots are taken before upgrades.
2. Delete the data directory and resync from peers. Works, but it takes the node out of participation for the whole sync and puts sustained read load on the three remaining validators. On a 1.4 TB directory this is hours to days, and it must not overlap with any other maintenance.
3. Copy a data directory from a peer still on the old release. Fast, but the copy must be from a stopped or snapshotted node, must be the same release and format, and the key file must not be carried over - otherwise two nodes share one identity.
4. Stay on the new release and fix forward. If the suspected regression is a round-change blip rather than a data-integrity problem, evidence-gathering on the new release may be cheaper than a rollback. That decision should be made from logs and metrics, not from a suspicion.

Deleting the data directory on all four validators the same evening would remove the quorum (3 of 4 is required) and halt the chain. That proposal must be rejected outright.

The quorum rule drives everything: with four validators, quorum is 2F + 1 = 3, so exactly one validator may be unavailable at a time.

Pre-upgrade

1. Read the release notes for database, storage-format, consensus, and key-handling changes, and for any behaviour that requires all nodes to be on the same version to interoperate.
2. Rehearse on a disposable dev network with the same genesis and block period.
3. Take a filesystem snapshot of each validator immediately before its upgrade, and record the block height. This is the rollback path - without it, rollback means resync.
4. Confirm no other change is in flight: no validator set vote, no genesis-related change.

Per-validator sequence

1. Stop one validator, confirm the remaining three still produce blocks (watch head height for at least several blocks).
2. Start it on the new release, watch startup migration complete, and confirm the genesis hash is unchanged and the chain ID matches.
3. Gate on the node reaching the network head, joining rounds, and being seen by peers before touching the next validator. A node that starts but is 200 blocks behind is not "done".
4. Repeat, one validator at a time, never two.

During - alert on head-height divergence, round-change rate, and commit counts. The round changes seen under load are exactly the signal to investigate before proceeding to the next node, and to capture diagnostics from the upgraded node while it is still the only one upgraded. That is the cheapest moment to decide fix-forward versus rollback, and it is why the snapshot matters more than the rollback tooling.

During sync

• Watch eth_syncing and the head-height metric on v2, and disk I/O on the three serving validators - a 1.4 TB resync against a three-node serving set can slow block import on the producers. Throttle or schedule accordingly.
• Watch v2's disk capacity. A snap sync needs room for both the downloaded state and the temporary data; running out at 95% wastes the whole attempt.
• Do not restart v2 mid-sync to "see if it helps"; repeated restarts can reset the pivot and never converge.
• Confirm the genesis hash on v2 matches a validator before the sync starts. A wrong genesis syncs nothing.

Acceptance before calling it a participant

1. Head height equals the network head, not merely close to it, and stays equal for several blocks.
2. net_peerCount is stable and the validators list the node as a peer.
3. The node's validator address is the one in the genesis validator list - resyncing does not change validator membership, so no vote is required, but the key must be the original validator key and not a regenerated one.
4. The node is observed committing: check logs for its own commit or prepare messages in rounds where it is the proposer.
5. Monitoring and backups are re-attached to the node, and the snapshot taken before the incident is retained until the next successful upgrade completes.

Only then resume using it as the upgrade test bed for the remaining validators - one at a time.