Deploying a Consul Datacenter
Bootstrap and join
A server config sets server = true, a unique node_name, a persistent data_dir, and bootstrap_expect = 3 (or 5). bootstrap_expect tells the first servers how many peers to wait for before electing a leader; it is a cluster-formation hint, not a security setting, and it must be the same on every server. The legacy -bootstrap flag is deprecated and should never appear in production answers.
Joining is done with retry_join, which should list addresses (or provider=aws tag_name=... cloud auto-join blocks) so agents retry until the cluster is reachable. Client agents point retry_join at the server pool; servers point at each other. -join on the command line joins once and does not retry, so it is the wrong answer for automated deployment.
Scaling safely
Add or remove servers one at a time and let Raft catch up (consul operator raft list-peers) before touching the next one. Replacing all servers simultaneously destroys quorum. Use leave_on_terminate for cloud autoscaling and skip_leave_on_interrupt deliberately.
Traps
- Changing
bootstrap_expectafter bootstrap does nothing useful; it is only read when the cluster forms. retry_joinneeds addresses reachable on port 8301 (gossip) and 8300 (RPC); an agent that can ping but not reach 8301 will never join.- Datacenter name and encryption keys must match across agents in the same datacenter.
Rule of thumb
For a production cluster: 3 servers, bootstrap_expect = 3, identical encrypt key, retry_join on every node, TLS configured before the cluster is exposed.