I85PREP
⌘ K
Consul Ops Pro study materials
Consul Ops Pro study resources

Consul Ops Pro: Observability, Monitoring, and Logging

Consul Ops Pro · Work through your exam, one topic at a time.

All guide sections

Observability, Monitoring, and Logging

Domain 6 asks how you know a Consul cluster is unhealthy before users tell you it is. Expect questions on which endpoint exposes which signal, how telemetry is configured, how log levels behave, and what is worth alerting on.

Where the signals live

  • Metrics: every agent exposes JSON at GET /v1/agent/metrics. For Prometheus scraping, set telemetry { prometheus_retention_time = "24h" } and scrape /v1/agent/metrics?format=prometheus; otherwise metrics must be pushed to StatsD, Statsite or DogStatsD via the telemetry stanza.
  • Health endpoints: /v1/status/leader, /v1/agent/self, /v1/agent/members, /v1/health/service/<name>.
  • Logs: log_level accepts TRACE, DEBUG, INFO, WARN, ERR; log_file and log_rotate_bytes handle file output; consul monitor -log-level=debug streams logs from a running agent.
  • Diagnostic bundles: consul debug -duration=2m captures logs, metrics, pprof profiles and member lists into an archive for escalation.

Metrics that matter

Watch consul.raft.leader.lastContact (leader staleness - rising values mean CPU or network trouble), consul.raft.apply (log write rate), consul.raft.commitTime (disk and fsync latency), consul.serf.member.flap (nodes cycling), and consul.runtime.alloc_bytes (GC pressure). Alert on: no leader for more than a few seconds, sustained commit latency, member flapping, and licence expiry in Enterprise.

Enterprise and mesh observability

Audit logging (audit { enabled = true, sinks = [...] }) records every API operation with the token that performed it - the way you prove who deleted a KV key. Connect visibility comes from Envoy: enable the admin access log path when generating proxy config, scrape Envoy stats separately, and use the Consul UI topology view to see service-to-service edges and intention status.

Worked example

Users report intermittent 503s across a three-datacenter mesh, and consul.serf.member.flap is spiking in one datacenter. Reasoning: flapping means the gossip layer keeps losing contact, so those nodes' serfHealth goes critical, their services drop out of DNS and proxy upstream sets, and clients see 503s. Investigate 8301 UDP reachability, MTU and CPU starvation before touching intentions - the fault is at the membership layer, not the authorization layer.

Traps

Enabling the Prometheus format but forgetting prometheus_retention_time, so scrapes return nothing; querying a client agent for Raft metrics and concluding the cluster is idle; leaving DEBUG on in production; and alerting only on individual service checks while never alerting on leader presence.