Stale bootstrap_expect plus empty data directories: no cluster leader

consul members reports Serf LAN membership, a gossip layer entirely separate from Raft. All three agents can gossip, so all three show alive regardless of Raft health.

Raft needs a persisted configuration or an initial bootstrap. Because the instances were rebuilt with empty data directories, there is no on-disk Raft configuration. In that state a server self-bootstraps only when it observes bootstrap_expect peers. All three configs say 5, only 3 exist, so every server waits for two servers that will never appear and none creates an initial configuration. The log line raft: no known peers, aborting election is the fingerprint of an empty Raft configuration, not of quorum loss (which produces repeated election timeouts with known peers).

So the root cause is compound: loss of Raft state + a bootstrap_expect value that no longer matches the real voter count. Either condition alone would not have produced this outage.

Path A: relabel and restart (fresh cluster). Set bootstrap_expect = 3 on all three servers and restart. This forms a new empty cluster: all KV data, intentions, ACL policies and tokens, and service registrations disappear. With ACLs enabled, a fresh cluster with no seeded management token is also unusable until you supply acl.tokens.initial_management or re-bootstrap ACLs. Acceptable only if the snapshot is untrustworthy.

Path B: restore the 14-hour-old snapshot (recommended).

1. Stop all Consul agents.
2. Start one server alone with bootstrap_expect = 1 (or -bootstrap-expect=1); wipe its data directory first if it holds partial state.
3. Wait for a leader, then run consul snapshot restore backup.snap with a management token that exists in the snapshot.
4. Verify consul operator raft list-peers shows one voter and consul kv get returns expected keys.
5. Start the other two servers with bootstrap_expect = 3; they join and receive the restored state. Revert the temporary bootstrap_expect = 1 and restart the first server one last time.
6. Confirm three voters, one leader, autopilot healthy, ACL token list populated, and that client agents have re-registered services.

Data loss is limited to roughly 14 hours of KV writes and catalog changes; agent anti-entropy re-publishes registrations automatically. Never leave -bootstrap in a production config: with persistent state an accidental -bootstrap can create an isolated single-node cluster.

• Derive bootstrap_expect from inventory, not a static literal. Render it from the same source that defines server count, and add a pre-flight assertion in the DR runbook that consul operator raft list-peers voter count equals bootstrap_expect.
• Alert on the actual failure mode. Page on consul.raft.state.leader == 0 sustained, on raft: no known peers log matches, and on disk usage of the data directory (the original trigger).
• Protect the data directory. Move /var/consul to a dedicated volume with log rotation capped, and alert at 80% utilization rather than discovering corruption.
• Keep snapshots externally and verify them. Store in S3 with retention, run consul snapshot inspect on each new file, and rehearse a restore quarterly into a scratch environment.
• Use Autopilot intentionally. With cleanup_dead_servers and server_stabilization_time configured, dead voters are removed automatically, so the voter count and bootstrap_expect do not drift apart silently.