Consul Ops Pro case studies
Consul Ops Pro · Put your knowledge to work in context.
Consul Ops Pro is an independent study track for engineers who run HashiCorp Consul in production rather than merely consuming it. It targets platform, SRE, and infrastructure engineers responsible for cluster design, server lifecycle, service mesh rollout, ACL and TLS posture, monitoring, and recovery. The Consul Ops Pro case studies are 5 cases and about 3,992 words.
1dc3 joins the wrong gossip pool after a rebuild
Acme runs Consul 1.16.4 in three datacenters: dc1 (primary, 5 servers), dc2 (3 servers) and dc3 (3 servers). ACLs and Connect are enabled in all three, and agents use TLS for RPC with verify incoming and verify outgoing; verify server hostname is still disabled pending a certificate cleanup.
858 words, about 4 min2Stale bootstrap_expect plus empty data directories: no cluster leader
Northwind's production cluster began as five Consul servers with bootstrap expect = 5 hard-coded in a static configuration file. Two servers were decommissioned six months ago using consul leave and consul operator raft remove-peer , leaving three voters, but nobody edited the configuration.
756 words, about 4 min3Catalog-API registrations leave dead payment instances in DNS
Payments runs 60 instances in an Auto Scaling group. A deployment script registers each instance by calling PUT /v1/catalog/register on a server's HTTPS API, using an ACL token with service:payments:write and node:write on the instance's node name.
729 words, about 4 min4Gossip key rotation leaves a rack and three servers locked out
Contoso has 7 servers and 45 client agents, all with encrypt = "K1" in their configuration files; gossip encryption is the only encryption in use. The security team requires rotation to K2.
789 words, about 4 min5Expired snapshot token and a scoped KV wipe: full restore or surgical import?
Fabrikam runs a three-server Consul cluster with ACLs, Connect and 3,000 keys under config/ . Nightly at 02:00 a systemd timer runs consul snapshot save using a token issued by Vault with a 24-hour TTL; the timer's exit code is not monitored.
860 words, about 4 min