Consul Ops Pro: Cluster Bootstrapping, Server Management, and Upgrades

Domain 2 covers the lifecycle of server agents: forming the initial Raft cluster safely, managing membership afterwards, and upgrading servers without losing quorum. Questions here are usually "what happens next" scenarios - a second server is added to a cluster configured with bootstrap_expect = 3, or an operator restarts all three servers at once to apply new configuration.

Bootstrapping rules

• bootstrap_expect must be identical on every server in the datacenter and must equal the number of servers you will actually run. Servers join the gossip pool, learn about each other, and elect a leader only once the expected count is present.
• Do not confuse it with the deprecated -bootstrap flag, which self-elects immediately and risks split brain.
• Every agent must share the same encrypt gossip key from the outset; consul keygen produces it. Keys can be rotated later with consul keyring, but a node that never had the key can never join.
• Prefer retry_join (or cloud auto-join) over a one-shot -join, so restarted nodes rejoin by themselves.

Autopilot and membership

Autopilot, enabled by default in current versions, adds servers to the voting set only after they have been stable for server_stabilization_time (default 10s) and have caught up on the Raft log, and it prunes dead servers. Inspect with consul operator raft list-peers and consul operator autopilot get-config. Remove a permanently gone server explicitly with consul operator raft remove-peer; never by deleting its data directory and hoping.

Upgrades and worked example

A five-server cluster must move to the next minor version. Correct order: take a consul snapshot save first, upgrade servers one at a time with non-leaders first, and after each restart wait until consul members shows the node alive and raft list-peers shows it as a voter with a caught-up log. Then move leadership off the old leader with consul operator raft transfer-leader and upgrade it last, followed by clients. Reasoning: a five-voter Raft group tolerates exactly one voter being unavailable, so serialising restarts preserves quorum; moving the leader last avoids an election storm in the middle of the change.

Traps

Changing bootstrap_expect after bootstrapping; restarting every server simultaneously; assuming an easy downgrade after a major version bump; forgetting that consul leave removes a server from the Raft configuration, so a replacement node needs to rejoin explicitly; and skipping the pre-upgrade snapshot.