Consul Ops Pro cheat sheet

Consul agents run in either server or client mode; servers form the Raft quorum and maintain cluster state, while clients forward RPCs to servers and run local health checks.

A datacenter is the basic unit of Consul deployment; each datacenter has its own independent Raft group and gossip pool, and cross-datacenter communication occurs through the WAN gossip pool and RPC forwarding.

The gossip protocol (LAN and WAN) uses Serf and is responsible for membership, failure detection, and event broadcast; it is separate from Raft consensus used for state replication.

Raft requires a quorum of (N/2)+1 servers; a 3-server cluster tolerates 1 failure, a 5-server cluster tolerates 2 failures, and larger clusters increase write latency without proportional benefit.

Client agents are stateless except for local health check state; they can be scaled horizontally and are typically deployed on every node that hosts services.

Consul supports multiple datacenters with a single ACL and intention configuration when using enterprise features like replication; in open source, each datacenter must be configured separately.

Network segmentation: servers should be placed in a dedicated fault domain, and ports 8300 (RPC), 8301 (LAN gossip), 8302 (WAN gossip), 8500 (HTTP), and 8501 (HTTPS) must be allowed between appropriate components.

For production, deploy at least 3 servers (5 recommended for large clusters) across availability zones, and use client agents on all service nodes; avoid running servers on the same physical host as critical clients.

Consul on Kubernetes uses a Helm chart that deploys servers as a StatefulSet and clients as a DaemonSet; the connect-injector handles sidecar injection for service mesh.

Bootstrap a new cluster with consul operator raft bootstrap or by starting the first server with -bootstrap-expect 3; the latter waits until the expected number of servers join before electing a leader.

Use consul operator raft list-peers to view the current Raft configuration and consul operator raft remove-peer to remove a failed server that is not gracefully leaving.

To add a server, start it with the same gossip encryption key and -retry-join pointing to an existing server; then use consul operator raft add-peer or rely on automatic joining if -bootstrap-expect is not set.

Graceful server removal: use consul leave on the server being removed; it will transfer leadership if it is the leader and then shut down.

Upgrades should be performed one server at a time, waiting for the cluster to become healthy (consul members shows all alive and consul operator raft list-peers shows a stable leader) before proceeding to the next server.

Consul supports rolling upgrades across minor versions; always upgrade servers before clients, and ensure the new version's protocol is compatible (check the upgrade guides for version-specific notes).

Snapshot save and restore: consul snapshot save backup.snap and consul snapshot restore backup.snap; snapshots include KV, ACLs, intentions, and service catalog but not the Raft configuration.

Autopilot (enabled by default) performs health checks on servers, stabilizes the Raft cluster after disruptions, and can automatically remove failed servers when cleanup_dead_servers is true.

To change the cluster's encryption key, use consul keyring commands: install new key on all servers and clients, then remove the old key; gossip encryption keys must be identical across all agents in a datacenter.

Services can be registered via configuration files, the HTTP API (/v1/agent/service/register), or by using consul services register; registrations are persisted in the catalog and survive agent restarts if defined in config files.

Health checks can be script, HTTP, TCP, TTL, Docker, or gRPC; a service is considered healthy only if all its checks pass, and the DNS interface returns only passing instances by default.

TTL checks require periodic updates via consul services check or the /v1/agent/check/update endpoint; if the TTL expires, the check becomes critical.

Connect enables service mesh with sidecar proxies; enable it with connect { enabled = true } in the agent config, and register a service with a sidecar proxy using connect { sidecar_service {} }.

Intentions define allowed connections between services; they are deny by default when ACLs are enabled, and can be managed via API, CLI (consul intention), or configuration entries.

Service defaults and service intentions can be configured centrally using configuration entries (e.g., service-defaults, service-intentions) for consistent mesh behavior.

Health check definitions support interval, timeout, deregister_critical_service_after, and status; the latter can set an initial status to avoid false alarms during startup.

For service mesh, sidecar proxies can be deployed manually or automatically via consul connect envoy command; Envoy is the default proxy, and its configuration is generated by Consul.

Service tags and meta can be used for filtering and routing; they are included in DNS SRV records and API responses, and can be used in intentions for finer-grained control.

The KV store is a hierarchical key/value store accessible via HTTP API (/v1/kv/), CLI (consul kv), and DNS with TXT records; keys are case-sensitive and support atomic operations via CAS (Check-And-Set).

KV entries have associated flags, create/modify indices, and optional session binding for locking; sessions enable distributed locks with TTL and health checks.

ACLs are enabled by setting acl { enabled = true } on all servers; the bootstrap token is generated with consul acl bootstrap and has unrestricted privileges.

ACL policies are composed of rules that grant read, write, or deny on resources like service, node, key, query, intention, and operator; policies are attached to tokens.

Token types: management tokens (global access), client tokens (limited by policies), and anonymous tokens (used when no token provided); the anonymous token policy should be minimal.

Intentions are defined with source and destination service names and an action (allow/deny); they are evaluated in order of precedence: exact match > prefix match > wildcard, and deny overrides allow.

ACL replication in enterprise: primary datacenter replicates ACL policies, tokens, and intentions to secondary datacenters; in open source, each datacenter must be configured independently.

Use consul acl token create with -policy-name or -policy-id to create tokens; tokens can also be set via the CONSUL_HTTP_TOKEN environment variable or -token flag.

KV locks: create a session with consul session create, then acquire a lock with consul kv put -acquire=<session> key value; release with -release or when session expires.

Gossip encryption uses a symmetric key (32 bytes base64-encoded) configured via encrypt in the agent config or -encrypt flag; all agents in a datacenter must use the same key.

TLS for RPC and HTTP: configure verify_incoming, verify_outgoing, and verify_server_hostname on servers; clients need verify_outgoing and CA certificate to verify servers.

Certificates must include server.<datacenter>.<domain> in the SAN for server verification; use consul tls cert create to generate certificates with proper SANs.

Enable HTTPS for the HTTP API by setting ports { https = 8501 } and providing cert_file and key_file; the HTTP port can be disabled or redirected.

ACL tokens should be used to protect the KV store and service registration; use acl { tokens { default = "..." } } to set a default token for an agent.

Secrets management: Consul can store secrets in the KV store, but for production, integrate with Vault using the Consul secrets engine or use Vault to manage Consul ACL tokens.

The consul tls ca create command creates a CA and consul tls cert create -server creates a server certificate; distribute the CA certificate to all agents.

To rotate gossip keys, use consul keyring -install <new-key> on all agents, then consul keyring -use <new-key>, and finally consul keyring -remove <old-key>.

TLS can be used for the Connect service mesh with connect { ca_provider = "consul" } or Vault; the built-in CA is not recommended for production multi-datacenter.

Consul exposes metrics in Prometheus format at /v1/agent/metrics and via telemetry configuration; key metrics include consul.runtime.alloc_bytes, consul.raft.apply, consul.serf.member.flap, and consul.http.*.

Enable telemetry with telemetry { prometheus_retention_time = "1m" } and optionally dogstatsd_addr for StatsD; metrics can be scraped by Prometheus.

Logs are written to stderr by default; set log_file and log_level in the agent config; levels are TRACE, DEBUG, INFO, WARN, ERR.

Health checks generate events and can trigger alerts; use consul watch to monitor changes and integrate with monitoring systems.

The /v1/status/leader and /v1/status/peers endpoints provide cluster health information; consul operator raft list-peers shows Raft configuration.

Distributed tracing for Connect can be enabled with connect { enable_central_service_config = true } and configuring Envoy tracing; Consul supports Jaeger and Zipkin.

Audit logging (enterprise) logs all API requests with tokens and can be configured to write to a file or syslog.

Use consul monitor to stream logs from a running agent at a specified log level; useful for debugging without restarting.

Key alerts: consul.raft.leader.lastContact > 200ms indicates leader contention; consul.serf.queue.Intent > 0 indicates gossip issues.

If a server loses quorum, it cannot process writes; recover by restoring a snapshot or by using consul operator raft recover to force a new cluster from a peer's data.

To recover from a lost quorum, identify a server with the most up-to-date data, stop all servers, and restart that server with -bootstrap-expect 1 and -bootstrap to form a new cluster, then rejoin others.

Snapshot restore must be performed on a stopped server or on a new cluster; use consul snapshot restore with the -http-addr of a running server.

Common issues: port conflicts (check consul members and firewall), TLS misconfiguration (check certificate SANs and verify_outgoing), and ACL token missing (check consul acl token list).

Use consul debug to collect a debug bundle including logs, metrics, and configuration for support.

If a client agent cannot connect to servers, verify retry_join and network connectivity; consul info shows agent status and consul members shows cluster membership.

For a failed server, remove it from Raft with consul operator raft remove-peer -id <node-id>; then start a new server with the same configuration.

Backup strategy: take regular snapshots (e.g., every 24 hours) and store them off-cluster; test restore procedures periodically.

DNS issues: ensure recursors are configured if using Consul for external DNS; check consul catalog nodes to verify service discovery.

Consul integrates with Terraform via the Consul provider to manage services, KV, ACLs, and intentions as code; use terraform plan and apply for changes.

The Consul Helm chart supports declarative configuration of servers, clients, and Connect injector; values.yaml can be version-controlled.

Nomad integration: Consul is used for service discovery and Connect; Nomad job specifications can register services and health checks automatically.

Enterprise features include: automated backups, redundancy zones, read replicas, audit logging, namespaces, and SSO/OIDC authentication.

Namespaces (enterprise) allow multiple teams to share a cluster with isolated service and KV namespaces; ACLs can be scoped to namespaces.

Admin partitions (enterprise) provide larger-scale isolation, allowing multiple datacenters to be managed as a single unit with separate Raft groups.

Consul's API can be automated with CI/CD pipelines; use consul services register in deployment scripts or the HTTP API.

Vault integration: use Vault to manage Consul ACL tokens and certificates; the Consul secrets engine dynamically generates tokens.

Consul on Kubernetes can be automated with the official Helm chart and the Consul K8s CLI; use consul-k8s install for a quick setup.