New Datacenter Never Elects a Leader

Auto-join success only proves the IAM call to ec2:DescribeInstances returned addresses. It says nothing about whether gossip or RPC on those addresses is reachable, which is the classic trap here.

Diagnostic order:

1. consul members on each server. A node that lists only itself has no LAN gossip peers - that points at the network, not at Raft. If a node lists peers but they are marked failed, you are looking at a different problem.
2. Agent log for the join error text. dial tcp <ip>:8301: i/o timeout means the security group is blocking; encryption key mismatch means the encrypt key differs; a log line naming bootstrap_expect points at the template fault. The three lines have completely different remedies, so read before changing.
3. Verify the actual port matrix on the cross-AZ path: 8300/tcp (server RPC), 8301/tcp and 8301/udp (LAN gossip), 8302/tcp and 8302/udp (WAN gossip), and 8600 for DNS where clients query directly. The scenario has cross-AZ rules only on the client security group, so server-to-server gossip across AZs is blocked - and with one server per AZ, no three servers can see each other, which alone explains "no cluster leader".
4. Diff bootstrap_expect across all five config files. A mismatched value is a second, independent defect that will keep the cluster from converging cleanly even after the networking is fixed.

Fix the network first, then normalize every server's config to the same bootstrap_expect, then restart one at a time and watch for a leader before touching the clients.

bootstrap_expect declares how many servers should participate in Raft before the cluster considers itself formed, and it must be identical on every server. Here the intended steady state is five servers, so it should be 5, with quorum of 3 and tolerance of two failures.

Consequences of the values in the configs:

• bootstrap_expect = 3 on a five-node deployment: the first three servers to see each other will form a cluster and elect a leader immediately, which masks the missing cross-AZ connectivity during rollout. The remaining two eventually join as voters, silently taking the peer set from three to five and moving quorum from 2 to 3. You end up with the right number of voters but no deliberate step where that happened, and the initial three-server window gives you a false sense of a working deployment.
• bootstrap_expect = 5 on only one node: the values disagree, so the cluster does not converge consistently and the divergence shows up as odd, partial join behaviour rather than a clean error. Treat any mismatch as a defect to fix before further debugging.

Also: never set bootstrap and bootstrap_expect together on the same agent - they conflict. For a cluster this size, the durable answer is a single templated config rendered from one value, so a rebuilt box cannot reintroduce a stale setting, and Autopilot's steady-state health checks are then meaningful because the expected server count never moves underneath them.

Gossip encryption operates on the Serf layer, which is separate from RPC TLS and from Connect mTLS. Members must share the same key: a client with a different key fails the handshake and never appears in the gossip pool, so the server never learns about it and it can never register services - matching the 'all 60 clients failed' symptom, and it would persist even after the security group is fixed.

Rollout:

1. Put the same encrypt value in every server and client config, across both VPCs. The WAN gossip pool also uses gossip encryption, so if this datacenter later federates with another, both sides need the same key.
2. Set encrypt_verify_incoming and encrypt_verify_outgoing to true once every agent is on the key. Leaving verification off means an agent accepts unencrypted gossip, which defeats the purpose.
3. Restart clients in batches and confirm each appears in consul members before moving on.

Rotation, later, without downtime:

• consul keyring -install <newkey> on every agent (installs without using it),
• consul keyring -use <newkey> to make it primary,
• consul keyring -remove <oldkey> after all members are on the new key.

Note the boundary: gossip encryption protects Serf traffic only. It does nothing for RPC or HTTP, which need verify_incoming/verify_outgoing/verify_server_hostname, and it is unrelated to Connect's mTLS. Also remember the gossip key is agent configuration, so it is not captured in a Consul snapshot and must be stored in your secrets system alongside the restore runbook.