Consul Associate common mistakes
Consul Associate · Understand the traps and how to avoid them.
Treats the LAN gossip pool and WAN gossip pool as interchangeable - e.g., opens only TCP/UDP 8301 between servers in different datacenters and expects cross-DC federation to work, or assumes client agents must join the WAN pool.
Candidates memorize a single 'gossip port' and never separate the pools: 8301 carries LAN Serf (servers plus clients inside one datacenter), 8302 carries WAN Serf (servers only, across datacenters), 8300 is server RPC, and 8600/8500/8501 are DNS/HTTP/HTTPS. Federated lookups and cross-DC prepared-query failover silently break when 8302/8300 are blocked between servers.
What to do instead
Learn the port matrix by pool and by role: within a datacenter every agent needs 8301 TCP+UDP to the other agents and 8300 to servers; between datacenters only servers need 8302 (WAN gossip) and 8300 (RPC). Verify with consul members -wan versus consul members before blaming configuration.
Shrinks a running datacenter by editing bootstrap_expect downward and restarting servers, or gives different bootstrap_expect values to different servers.
bootstrap_expect only governs the initial bootstrap election and defines the quorum size (majority of the declared value). Mismatched values can let a single server bootstrap on its own and form a competing cluster, and a stale high value means too few live servers to reach quorum, so the cluster loses its leader and writes fail.
What to do instead
Never change bootstrap_expect as a scaling mechanism. Take a snapshot, then decommission servers one at a time with consul leave (graceful removal from the Raft peer set), confirming consul operator raft list-peers shows a leader and full health after each step; keep bootstrap_expect identical on every server and equal to the initial cluster size.
Uses `consul catalog services` or `consul catalog nodes` to decide whether a service is healthy and then reports healthy instances that are actually failing.
The catalog endpoints return registration data regardless of health state - a service with a critical check still appears there. Candidates conflate 'registered' with 'available' and miss the separate health subsystem.
What to do instead
Use consul health service <name>, consul health checks, the /v1/health/service endpoint with ?passing, or DNS lookups for availability decisions. State explicitly that the catalog stores intent-to-register while the health endpoints reflect liveness and passing status.
Queries a prepared query at `<name>.service.consul` (or via the catalog endpoint keyed by tag) instead of `<name>.query.consul`, and then concludes prepared queries are not resolving.
Prepared queries live in a separate namespace from plain service lookups. They are addressed at <name>.query.consul over DNS or via /v1/query/<id-or-name>/execute, and template queries need the template name supplied at query time, so the familiar service name format returns nothing useful.
What to do instead
Remember the three DNS forms: <service>.service.<dc>.consul for direct service discovery, <name>.query.consul for prepared queries, and <node>.node.<dc>.consul for node lookups. Create the query first (consul query create), confirm with consul query list, and configure failover (nearest N datacenters) or the query returns empty rather than falling back.
Stores passwords, API keys, or other secrets in the KV store and treats it as a secret manager.
It is easy to assume that because Consul supports ACLs and TLS in transit, the KV store also encrypts data at rest. It does not: values are stored in plaintext in the Raft log and snapshot files on every server, visible to anyone holding a key:read policy on that prefix.
What to do instead
Keep only configuration data in KV. Use Vault (with its Consul backend or templated sync) for secrets, or encrypt values client-side before writing. Design ACL policies at prefix granularity so read access to a prefix is itself a security decision.
Enables ACLs with the default deny policy without provisioning tokens for agents, DNS clients, and the UI, then reports that DNS lookup and service discovery 'stopped working'.
With ACLs on and default deny, the anonymous token has no privileges, so unauthenticated DNS queries and HTTP requests return no results, and agent-level operations (anti-entropy sync, checks, UI) fail. Candidates test only with a management token from the CLI and never test the unauthenticated path.
What to do instead
Plan tokens before flipping the switch: set acl.tokens.default (and acl.tokens.agent/config equivalents) on each agent, pass ?token= for DNS/HTTP, and either grant the anonymous token a read-only node/service policy for public discovery or document that all clients need tokens. Re-test DNS with and without a token after enabling.
Loses the bootstrap ACL token (or never captures it) after running `consul acl bootstrap`, and then cannot administer the cluster.
Bootstrap succeeds exactly once per cluster and the secret is printed only at that moment. Candidates run it, close the terminal, and discover later that no management token exists; there is no second chance without a deliberate reset.
What to do instead
Capture and vault the SecretID immediately. For reproducibility, pre-seed acl.tokens.initial_management (Consul 1.11+) in the server config. If already lost, remove the acl-bootstrap-reset file from the leader's data directory and restart that server to re-enable bootstrap, then re-bootstrap once and store the token securely.
Rotates the gossip encryption key by running `consul keyring -install` and then immediately `consul keyring -remove` on the old key, or installs the new key on only some agents.
Key changes are a multi-phase, cluster-wide operation. Removing the old key while any agent still only knows it ejects those agents from the gossip pool; installing on a subset makes the pools inconsistent and produces 'failed to decrypt' / member-flap errors.
What to do instead
Use the three-phase order: install the new key on every agent (consul keyring -install), verify all members see both keys (consul keyring -list), switch the primary with -use on the servers, wait for propagation, then -remove the old key only after every member reports the new key as primary.
Assumes intentions police all service-to-service traffic by default, or that the default intention is deny, and then claims a service is 'blocked by Connect' when it is still reachable.
Intentions are enforced only for traffic that actually traverses a Connect sidecar proxy. Anything hitting the service's own port directly bypasses them, and the built-in default intention is allow, so a cluster with proxies deployed but no intentions written permits everything.
What to do instead
State the model explicitly: Connect must be enabled, traffic must flow through the proxy (or a gateway), and the default intention is allow unless you add a catch-all deny. Verify with consul intention check <source> <destination> and consul intention list rather than assuming, and remember L7 permissions are deny-by-default for paths that match no rule.
Treats `consul snapshot save` as a complete backup of the datacenter and expects `consul snapshot restore` to merge cleanly into an existing cluster with different membership.
A snapshot contains the Raft state (KV, catalog, ACLs, intentions, prepared queries, Connect CA) but not agent configuration files, TLS certificates, gossip keys, or the license - and restore replaces the whole state on the target rather than merging. Candidates also run restore against a non-leader or onto a cluster whose membership they expect to be preserved.
What to do instead
Back up configuration, certificates, and gossip keys alongside snapshots, take snapshots on a schedule from a server, and test restores into a scratch cluster. Run restore against the leader with an operator:write token, expect existing ACLs and tokens to be overwritten, and re-verify membership and consul operator autopilot health afterward.
Assumes a newly added server immediately counts toward quorum, and assumes a failed server is automatically removed as soon as it goes unhealthy.
With Raft protocol 3 and Autopilot, new servers join as non-voters and are only promoted after they have been stable for server_stabilization_time. Dead-server cleanup only removes a failed server once a healthy replacement exists, so candidates see a 'stuck' cluster and start deleting peers manually.
What to do instead
Inspect state with consul operator raft list-peers (Voter column) and consul operator autopilot health, then wait for stabilization or tune with consul operator autopilot set. Do not hand-remove peers; add the replacement, confirm promotion to voter, and let cleanup_dead_servers retire the failed member.