Consul Ops Pro: Troubleshooting, Failure Recovery, and Backup/Restore

Domain 7 covers the failure modes Consul actually produces and the recovery procedures that work. The exam tests whether you can tell a membership problem from a quorum problem from an ACL problem, and whether you know exactly what a snapshot does and does not restore.

Triage order

1. consul members - are all nodes alive? Split membership points at gossip: 8301 UDP or a key mismatch.
2. curl /v1/status/leader - an empty string means no leader, so either quorum is lost or servers cannot reach each other on 8300.
3. consul operator raft list-peers - confirm voters, leader, and applied index parity.
4. consul monitor - leader-election loops or failed to decrypt messages usually name the fault directly.
5. ACLs last: a burst of 403s after a restart usually means the agent lost its token or acl_down_policy took effect.

Snapshots

consul snapshot save backup.snap writes a point-in-time archive; consul snapshot inspect shows index and size; consul snapshot restore needs a running cluster and overwrites its state - restore onto a fresh, quorum-intact cluster, never over live production. Snapshots contain KV, catalog, sessions, prepared queries, ACL policies and tokens, config entries, and Connect CA state. They do not contain the licence, the gossip encryption key, or agent configuration; keep those in your secret store and infrastructure code. The Enterprise snapshot agent automates periodic saves to S3/GCS/Azure with retention and is the supported way to get consistent scheduled backups.

Worked example: quorum lost

A three-server datacenter loses two servers, and the survivor will not elect itself. Recovery: stop the survivor, write raft/peers.json in its data directory listing only that server ([{"id":"<node-id>","address":"<ip>:8300","non_voter":false}]), restart it, delete peers.json, then join replacement servers with retry_join and let them catch up. Reasoning: Raft will not shrink itself without quorum, and peers.json is a startup-only override that discards uncommitted entries - which is precisely why a snapshot restore is preferred whenever a recent one exists.

Traps

Restoring a snapshot into a cluster with a different datacenter name; assuming consul leave on a failed node is harmless, when it rewrites the Raft configuration if quorum exists; recovering without first identifying which node had the highest applied index; and misdiagnosing a gossip key mismatch as a Raft bug.