Gossip key rotation leaves a rack and three servers locked out

Keyring state is runtime gossip state, not configuration. consul keyring -install/-use/-remove mutate the in-memory Serf keyring and propagate via gossip; nothing is written to disk or to encrypt. On restart, an agent reads encrypt = "K1" from its config file and attempts to join using K1.

Mistake 1: K1 was removed before every agent had been restarted with K2 in its config. Once -remove K1 propagates, any agent still using K1 can no longer decrypt gossip, and its join fails with exactly the message observed. The 18 rack-B clients were healthy only because their in-memory keyring had already switched primary to K2 via gossip; the restart discarded that.

Mistake 2: No configuration change was made at all. Even if K1 had been left installed as a secondary key, every future restart would silently fall back to K1, so the rotation would never actually complete and would depend forever on a retired key remaining installed.

The three servers hit the same problem: they restarted, read K1 from config, and could not talk to peers who only accept K2. The blast radius is bounded to restarted agents, which is why it looked like a rack-specific incident rather than a rotation failure.

Finish the rotation; do not roll the cluster back to K1. Rolling back means re-installing K1 everywhere and re-running the cycle, and it reopens the same window. Recovery is additive:

1. Re-install K1 as a secondary key on a healthy server that still has K2 primary: consul keyring -install K1. K1 is now accepted for decryption but is not primary, so the 21 locked-out agents can rejoin on their next attempt.
2. Confirm propagation with consul keyring -list on several agents: both keys present, K2 marked primary.
3. Change encrypt to K2 on all 52 agents' configuration via the secret store/template pipeline, not by hand. Treat it as a normal rolling change.
4. Rolling restart in batches, verifying after each batch that the agent is alive in consul members and that its keyring -list shows K2 primary and its config now contains K2.
5. Only when every agent's config is K2 and every agent has been restarted run consul keyring -remove K1 once. -list should then show only K2 cluster-wide.

If an agent cannot be restarted soon, K1 must remain installed; removing it again recreates the outage. Track stragglers explicitly rather than assuming gossip completed the job.

• Store keys in Vault and render encrypt through the same templating path as the rest of the agent config, so the config value and the intended key never diverge.
• Two-phase job with an inventory gate. Phase 1 installs and uses the new key. Phase 2 removes the old key only after a check across the entire inventory, executed agent-by-agent, asserting both the runtime keyring and the on-disk config reference the new key. A failed gate aborts and leaves both keys installed, which is the safe state.
• Idempotency and audit. Log the key fingerprint, not the key; require change approval for the remove phase specifically.
• Monitoring. Alert on Serf join failures and consul.serf.member.flap increasing, on WAN/LAN member counts dropping below expected, and on any agent whose keyring diverges from the majority. A periodic inventory job that compares each agent's keyring -list to the expected key IDs catches drift long before a restart does.
• Longer term, move to TLS with verify_incoming/verify_outgoing/verify_server_hostname for RPC and keep gossip encryption as defense in depth; rotating a symmetric gossip key remains an all-agents-touching operation, so it should be rare and fully scripted.