Consul Architecture and Core Concepts
Agents: client vs server
Consul is one binary that runs as an agent in either client or server mode, and every node in a datacenter runs an agent. Client agents are the data-plane edge: they hold local service registrations, run health checks, expose the HTTP (8500), DNS (8600) and gRPC (8502) interfaces for that node, and forward all reads and writes to servers over RPC. Server agents are the control plane: they participate in Raft, and only servers persist the catalog, KV store, intentions and ACL data. Client state is a cache kept fresh by anti-entropy sync, which is why a client never answers from durable storage.
Two gossip pools, one RPC port
LAN gossip (Serf, port 8301 TCP+UDP) provides membership and fast failure detection inside a datacenter. WAN gossip (8302) federates servers across datacenters. Server RPC is 8300; CLI/HTTP is 8500/8501. A datacenter (dc1 by default) is the unit of gossip pool, catalog and Raft cluster.
Traps to expect
- "Client agents store KV data" - false; KV lives on servers in Raft.
- Quorum is (N/2)+1: 3 servers tolerate 1 failure, 5 tolerate 2. Even-sized clusters add cost without adding tolerance.
- Gossip encryption and RPC TLS solve different problems; a gossip key does not protect RPC traffic.
- A DNS name like
web.service.dc1.consulembeds datacenter and domain; changingdomainconfig changes it everywhere.
Rule of thumb
Ask "where does this data live?" - answer: Raft on servers. Ask "who can answer this query?" - answer: any agent, from cache or by forwarding to a server.