Consul Ops Pro common mistakes
Consul Ops Pro · Understand the traps and how to avoid them.
Treating the gossip encryption key as a per-server setting and generating a fresh key on each node when bootstrapping, then wondering why members never join the LAN or WAN pool.
Candidates conflate the gossip key (shared symmetric key for serf LAN/WAN encryption) with the TLS CA or with the server's own certificates, and assume each agent can have its own. They also forget that encrypt must be identical across all agents in the same gossip pool, and that encrypt_verify_incoming/encrypt_verify_outgoing defaults changed across versions.
What to do instead
Generate one key with consul keygen, distribute the same value to every agent via encrypt (or -encrypt at bootstrap), and verify with consul keyring -list that all members show the same key on both LAN and WAN rings. Remember key rotation uses consul keyring -install/-use/-remove, not a config rewrite.
Bootstrapping a multi-server cluster by putting `bootstrap_expect` on only one server or by setting `bootstrap: true` on several servers, then accepting a cluster that either never elects a leader or splits into competing clusters.
Candidates read the single-node quickstart and carry bootstrap: true into production, or they misunderstand that bootstrap_expect is a cluster-wide expectation that any server may use to trigger the initial election. They also forget that a server started with bootstrap: true ignores bootstrap_expect and can form a one-node cluster.
What to do instead
Set bootstrap_expect = 3 (or 5) identically on every server, never set bootstrap: true in a real cluster, and bring servers up one at a time. Confirm with consul operator raft list-peers and consul members that exactly the expected servers are voters and one is leader.
Adding a new server by copying an existing server's data directory (or snapshot-restoring onto a new node) instead of joining it as a fresh voter, producing duplicate node IDs, stale raft state, or a cluster that refuses the new peer.
Candidates treat Consul servers like stateless replicas and assume cloning is faster or safer. In practice the data directory contains the node ID, raft log, and serf state tied to that identity, so a clone collides with the original.
What to do instead
Install Consul, configure it with the same encrypt, retry_join, and bootstrap_expect, start it with an empty data dir, and let it join and receive the raft log. Use consul operator raft add-voter (or autopilot) to promote it, and verify with consul operator raft list-peers.
Registering services with `consul services register` or the HTTP API but omitting the health check or pointing the check at the wrong address/port, so the service appears in the catalog but never becomes healthy and never receives mesh traffic.
Candidates focus on getting the service into the catalog and forget that Connect and DNS/HTTP discovery filter on passing health. They also use 127.0.0.1 in a check that runs from a different network namespace, or a TCP check against the wrong port.
What to do instead
Always pair registration with a check (TTL, TCP, HTTP, or script) that is reachable from the agent running it. Verify with consul catalog services, consul health service <name>, and consul members. For mesh, confirm the proxy is registered and the service has connect { sidecar_service {} } or an explicit proxy registration.
Confusing intentions with ACL policies: granting a broad ACL policy and assuming it authorizes service-to-service traffic, or writing an intention and expecting it to grant catalog or KV access.
Both are named as access controls and both mention services, so candidates assume one implies the other. In reality intentions govern Connect data-plane authorization between services (and are evaluated by the proxy), while ACLs govern API/catalog/KV access for tokens.
What to do instead
Model them separately: use consul intention create -allow <src> <dst> (or intention config entries) for L4/L7 service authorization, and ACL policies/tokens for who can read or write the catalog, KV, and intentions themselves. Test with consul intention check <src> <dst> and by exercising the API with the token.
Writing an ACL policy with a wildcard resource or an unanchored prefix and assuming it is least-privilege, then failing questions about why a token can read KV paths it should not.
Consul ACL rules use resource/segment/policy syntax where key_prefix and service_prefix match by prefix, and * in the wrong position broadens scope. Candidates also forget the default-deny posture once ACLs are enabled, and that the bootstrap token is the only one that can create the initial policies.
What to do instead
Use explicit key_prefix and service rules, test with consul acl token read -self and by attempting the denied operation, and keep the bootstrap token offline after creating a management token. Remember enabling acl.enabled without a bootstrap token locks you out.
Enabling TLS verify on servers before distributing the CA and client certificates to all agents, or mixing `verify_incoming`/`verify_outgoing`/`verify_server_hostname` settings, causing agents to fail gossip or RPC and the cluster to appear down.
Candidates enable the strictest TLS flags first and expect agents to negotiate automatically. They also forget that verify_server_hostname requires server certificates with the correct SANs (server.<dc>.<domain>), and that HTTPS for the API needs ports.https plus a client address/scheme.
What to do instead
Roll out TLS in stages: distribute the CA to every agent, enable verify_outgoing and verify_server_hostname first, confirm consul members is healthy, then enable verify_incoming. Use consul tls cert create with the right -server/-client/-dc flags so SANs match, and check consul info and agent logs for TLS handshake errors.
Reading only the service-level health endpoint and missing that a failing check is caused by a node-level check (serf health, node TTL) or by an anti-entropy sync problem, so the diagnosis points at the wrong layer.
Candidates jump to consul health service and see critical without tracing which check failed. Node-level checks (including serfHealth) mark the whole node unhealthy and remove all its services from discovery, and stale anti-entropy can leave the catalog out of sync with the agent.
What to do instead
Walk the layers: consul members for gossip, consul health node <node> for node checks, consul health service <svc> for service checks, then consul monitor and the agent log for the specific check output. Confirm the agent is syncing by comparing consul catalog services with the agent's local config.
Taking a snapshot with `consul snapshot save` but never testing restore, or restoring a snapshot onto a running cluster without stopping writes, then discovering the restore is incomplete or the cluster diverges.
Candidates treat snapshot save as sufficient backup and assume restore is a simple copy-back. In practice a restore replaces raft state and must be done carefully, and an untested snapshot may be corrupt or from a different cluster/version.
What to do instead
Schedule consul snapshot save (or Enterprise snapshot agent) on a cadence, store snapshots off-cluster, and rehearse consul snapshot restore in a staging cluster. Before restoring, quiesce writes, verify the snapshot with consul snapshot inspect, and confirm raft peers and leader after restore.
Hand-editing agent config on each server instead of driving it from a shared source, so upgrades and topology changes drift and `consul reload` produces inconsistent clusters; or upgrading servers all at once instead of one at a time.
Candidates underestimate how much Consul config must be identical across servers (encrypt, retry_join, ports, TLS) and treat upgrades as a fleet operation. Rolling upgrades require protocol compatibility and leader stability.
What to do instead
Manage config with version-controlled templates (Terraform, Ansible, or Consul's own config entries where applicable), keep shared values in a common file, and upgrade one server at a time, waiting for consul operator raft list-peers to show a stable leader and for autopilot health before proceeding. Use consul validate on config before reload.