Snapshot Restore That Rolls Back ACLs and Intentions

consul snapshot restore is a whole-state replacement, not a merge. The snapshot carries KV, the catalog, ACLs (tokens, policies, roles, bindings), intentions, sessions, prepared queries, and coordinate data. Restoring it resets every one of those to the moment the snapshot was taken. That is why three unrelated categories of change - KV cleanup, tokens, intentions - all rolled back together. The 36 hours of work after the snapshot no longer exists anywhere in Consul.

Immediate actions:

1. Stop restoring and take a fresh snapshot now so the current (rolled-back but consistent) state is captured before anyone makes further changes.
2. Recreate the ACL objects from their source of record - the IdP, Terraform state, or a change ticket. They cannot be recovered from Consul; the SecretIDs of the deleted tokens are gone, so clients holding them must receive new tokens, not re-enable old ones.
3. Recreate the two intentions and confirm with consul intention check -src <source> -dst <dest> that the intended allow/deny result is back, then verify real traffic rather than trusting the list output.
4. Re-run the KV cleanup after diffing against the pre-restore list, so the 4,000 keys are removed deliberately rather than assumed gone.

The process fix is a snapshot immediately before any bulk mutation, plus a rule that ACL and intention changes are recorded outside Consul so they can be replayed when a restore regresses them.

Four of five servers are live, so quorum (3) is intact and the cluster can serve reads and writes - this is a capacity and resilience problem, not an outage.

• Autopilot will usually handle the dead peer. With cleanup_dead_servers enabled (the default) the leader removes peers that have been unreachable past the contact threshold and stabilization window, so s4 should drop out of the Raft peer set without intervention. Confirm with consul operator raft list-peers and consul operator autopilot state.
• If it does not happen - cleanup disabled, or the dead node was itself the leader in a way that blocked the operation - remove it explicitly: consul operator raft remove-peer -address=<s4_address>.
• Then add the replacement with the same bootstrap_expect value as the rest of the servers, the same encrypt key, and the same retry_join configuration, and wait for Autopilot to promote it and reach a healthy five-server state.

Important boundary: the restore did not, and cannot, bring s4 back. Snapshots capture the state machine, not the health or identity of the machines running it. Also note that the sizing argument cuts the other way from the restore: while running on four of five servers, failure tolerance is one server, so the replacement should not be deferred casually.

Five properties drive the rehearsal design:

1. It clobbers the target. Never rehearse against a cluster that serves traffic. Restore into a scratch cluster sized for the data volume; the restore replays the whole state machine, so the target's existing catalog, KV, and ACLs are overwritten.
2. ACL state travels with the data. If the target has ACLs enabled, after the restore you are authenticated against the snapshot's ACL state, not the target's. Keep the source cluster's management token available for the rehearsal or you will restore yourself out of the cluster.
3. Version direction is one-way. A snapshot taken by a newer Consul cannot be restored into an older one. Rehearsing an upgrade should include restoring the pre-upgrade snapshot onto the old version, which works; the reverse does not.
4. Agent configuration is out of scope. Gossip encryption keys, TLS certificates and CA material, and join configuration live in agent config files, not the snapshot. A restore will not rebuild those, so recovery runbooks must cover them separately.
5. Measure the restore, not just its success. Time it against your recovery objective; a 200k-key KV plus a full catalog can take materially longer than a toy cluster, and the number that matters is the one measured on production-sized data.