Security: TLS, Gossip Encryption, and Secrets Management
Domain 5 covers the three cryptographic layers Consul exposes: gossip encryption for Serf traffic on 8301/8302, mutual TLS for RPC and HTTP, and Connect certificates for service-to-service traffic. Questions test ordering - what must exist before you enable verification - and what fails when a certificate is missing a SAN.
Gossip encryption
encrypt takes a 32-byte base64 key from consul keygen. The key must be present on every agent before it joins; a mismatch yields serf: failed to decrypt and the node never shows up in consul members. Rotation uses consul keyring -list, then -install <new>, -use <new>, -remove <old>, with encrypt_verify_incoming and encrypt_verify_outgoing temporarily false so both keys are accepted during the transition.
TLS for RPC and HTTP
Required settings are ca_file, cert_file, key_file, plus verify_incoming, verify_outgoing and verify_server_hostname. Server certificates must carry SANs for server.<dc>.<domain>, localhost, and any load-balancer names; client certificates need client.<dc>.<domain>. verify_server_hostname = true is what prevents a compromised node from impersonating the leader, but it fails hard when SANs are wrong. Enable ports { https = 8501 } and disable plaintext HTTP with ports { http = -1 } once every client speaks TLS.
auto_encrypt lets clients receive certificates from servers (auto_encrypt { tls = true } on clients, allow_tls = true on servers), and auto_config goes further, distributing configuration - both exist to avoid hand-copying material to ephemeral clients.
Connect CA and secrets
connect { ca_provider = "consul" } is the built-in CA; ca_provider = "vault" delegates intermediate issuance to Vault's PKI and is common where Vault already runs. Root rotation is a two-stage operation: install the new root as inactive with consul connect ca set-config, let leaf certificates cycle, then promote it.
Worked example
After setting verify_outgoing = true, servers log x509: certificate is valid for consul-server-1, not server.dc1.consul. Reasoning: outgoing verification validates the peer certificate against the expected server-name pattern, so the certificate must include server.dc1.consul as a SAN. Regenerate the certificate with the correct SANs and roll it server by server; do not disable verification to make the error disappear.
Traps
Generating a different gossip key per node; enabling verify_incoming on clients before they hold certificates; rotating the gossip key without the verify flags; and leaving the bootstrap ACL token in plaintext on disk instead of Vault.