I85PREP
⌘ K
Consul Ops Pro study materials
Consul Ops Pro study resources

dc3 joins the wrong gossip pool after a rebuild

Consul Ops Pro · Put your knowledge to work in context.

All case studies

dc3 joins the wrong gossip pool after a rebuild

Acme runs Consul 1.16.4 in three datacenters: dc1 (primary, 5 servers), dc2 (3 servers) and dc3 (3 servers). ACLs and Connect are enabled in all three, and agents use TLS for RPC with verify_incoming and verify_outgoing; verify_server_hostname is still disabled pending a certificate cleanup. After dc3's servers were rebuilt, operators see two symptoms. Running consul members on a dc3 server lists 11 members: the three local servers tagged dc3, plus all eight servers tagged dc1 and dc2. Running consul members -wan on dc1 lists dc1 and dc2 but never dc3. Sidecars in dc3 cannot obtain leaf certificates, tokens created in dc1 are rejected in dc3, and cross-datacenter prepared queries return no results. dc3's configuration is generated from the same template as dc1 and contains datacenter = "dc3", primary_datacenter = "dc1", the same encrypt key as dc1, and retry_join = ["provider=aws tag_name=consul-server"] with no port or datacenter filter. The consul-servers security group currently permits all traffic between tagged instances in all three VPCs.

1. Explain both symptoms. What is misconfigured, and why does LAN membership succeed while WAN federation, ACLs and Connect all fail?

Read the worked answer

What is actually happening

dc3's servers are joining dc1's LAN Serf pool on 8301, not the WAN pool on 8302. retry_join without an explicit port defaults to the LAN port; federation requires joining the WAN pool, normally via a WAN-specific join/auto-join entry targeting 8302. LAN Serf is not scoped by datacenter: an agent that dials a remote server on 8301 is simply accepted into that pool, and consul members then renders every member with its own dc tag. That is exactly the 11-member output. consul members -wan queries the 8302 pool, which dc3 never joined.

Why everything downstream fails

  • ACLs: replication from primary to secondary runs over the WAN link and requires acl { enable_token_replication = true } plus a replication token with acl:read on the primary. No WAN link means no replication, so dc1 tokens do not exist in dc3 and calls return 403.
  • Connect: leaf certificates are signed in the primary_datacenter. dc3 cannot reach the primary CA, so sidecars cannot complete certificate signing.
  • Prepared queries / WAN queries: they route over the WAN pool that was never formed.

Note the TLS interaction: had verify_server_hostname been enabled, this misconfiguration would have failed earlier and differently, with certificate is valid for server.dc3.consul, not server.dc1.consul, because server certificates are issued per datacenter. Fix the join target first, then enable hostname verification.

2. Give the remediation sequence you would run today, plus the verification for each step.

Read the worked answer

1. Fix configuration on all three dc3 servers before restarting anything.

  • Keep datacenter = "dc3" and primary_datacenter = "dc1".
  • Replace the portless auto-join with a WAN-targeted join to dc1 servers on 8302 (WAN join block or explicit :8302 targets in your version).
  • Keep the identical encrypt key.

2. Purge the bad membership. Restarting one dc3 agent is not enough: the other dc3 servers have already gossiped the dc1/dc2 members to each other and will re-propagate them. Restart all three dc3 servers with the corrected config, and on dc1 run consul force-leave for any dc3 node still listed in the dc1 LAN pool.

3. Verify in this order.

  • consul members on a dc3 server shows only the 3 dc3 servers (no cross-DC entries).
  • consul members -wan on dc1 shows all 11 servers with the correct three dc tags.
  • curl /v1/acl/replication?dc=dc3 reports replication enabled and a recent index.
  • consul acl token list in dc3 returns dc1-created tokens.
  • consul connect ca roots -datacenter=dc3 matches the dc1 root, and consul connect envoy -sidecar-for=<svc> -bootstrap succeeds for one dc3 service.

4. Then harden. Once the WAN pool is correct, enable verify_server_hostname and re-run the checks; a genuine federation problem will now fail loudly at TLS instead of silently splitting pools.

3. Which configuration and monitoring controls would you standardize so this cannot recur in a fourth datacenter?

Read the worked answer
  • One base template, per-DC overrides. primary_datacenter, encrypt, TLS and ACL settings come from a single shared source; only datacenter, node metadata and join targets vary. Render and lint with consul validate in CI, and add an assertion that no join target lacks an explicit WAN port or WAN block.
  • Make the mistake impossible at the network layer. Permit 8301 only inside a datacenter, 8302 only between datacenter server sets, 8300 only from approved operators. A cross-DC LAN join then fails fast rather than creating an 11-member pool.
  • Alert on WAN topology, not just process health. Export and alert on the count of WAN members per dc tag, on serf_wan member flapping, and on ACL replication lag. A simple check: every DC's expected server count appears in consul members -wan.
  • Turn on verify_server_hostname on servers so per-DC certificate mismatches surface immediately.
  • Gate rebuilds. Add a post-provision smoke test that a rebuilt server shows exactly its own DC in consul members and appears in consul members -wan before it is marked healthy in the load balancer rotation.