Monitoring, Logging, and Observability
Scope of this domain
Monitoring, Logging, and Observability tests whether you can instrument a system and alert on what users actually feel: metrics, logs and traces, golden signals, SLIs and SLOs, error budgets and alert design.
Core rules and concepts
- Three pillars: metrics (aggregatable time series), logs (discrete structured events), traces (request paths across services with spans and correlation IDs).
- Four golden signals: latency, traffic, errors, saturation. RED for services (rate, errors, duration); USE for resources (utilisation, saturation, errors).
- SLI is the measured indicator; SLO is the target, such as 99.9% of requests under 300 ms over 28 days; SLA is a contract with consequences. Error budget equals 1 minus the SLO.
- Alert on symptoms (SLO burn rate), not causes (CPU above 80%). Multi-window burn rates control noise: fast burn pages, slow burn creates a ticket.
- High-cardinality labels such as user ID or request ID explode metric storage; put those in logs and traces instead.
- Structured JSON logs with correlation IDs, centralised in a log platform; never log credentials, tokens or personal data.
- Tracing uses context propagation (W3C traceparent) and sampling, head-based for simplicity or tail-based to keep errors.
Worked example
The team is paged nightly because CPU exceeds 80%, while users report no errors. Reasoning: CPU is a cause metric and the threshold is arbitrary, so pages are noise. Replace it with a burn-rate alert on a 99.9% availability SLO: page when the one-hour burn rate exceeds 14.4 times (2% of the monthly budget in an hour) and open a ticket at a six-hour six-times burn. Keep CPU dashboards for diagnosis during incidents, not for waking people up.
Traps that cost marks
- Cause-based paging instead of symptom-based SLO alerting.
- Treating the SLO as the SLA, or setting SLOs nobody measures.
- Cardinality blowups from unbounded label values.
- Logging personal data or secrets to central storage.
- Dashboards with no owner and alerts with no runbook.