Monitoring, Observability, and Site Reliability Engineering
This domain tests whether you can instrument a system, alert on what users feel, and reason quantitatively about reliability: metrics, logs and traces, the SLI–SLO–error budget model, and incident practice.
Core rules
- Three signals: metrics (cheap to aggregate, cardinality-sensitive), logs (rich detail, expensive at volume), traces (request-scoped causality, sampled). OpenTelemetry gives vendor-neutral instrumentation and propagates trace context across service boundaries.
- Frameworks: golden signals - latency (separate success from error latency), traffic, errors, saturation. RED for request-driven services (rate, errors, duration); USE for resources (utilisation, saturation, errors).
- Reliability maths: SLI = good events ÷ valid events. SLO = target over a window (for example 99.9% over 28 days). SLA = contractual with consequences. Error budget = 1 − SLO.
- Alert on symptoms (budget burn), not causes (CPU). Multiwindow multi-burn-rate alerting pairs a long window that proves significance with a short window that proves the problem is still happening.
- Blameless postmortems: timeline, contributing factors, owned and dated action items, shared widely.
Worked example
SLO 99.9% over 30 days. Budget = 0.001 × 30 × 24 × 60 = 43.2 minutes. A dependency degrades to a 1% error rate for six hours. Burn rate = error rate ÷ (1 − SLO) = 1% ÷ 0.1% = 10×. At that rate the budget exhausts in 30 ÷ 10 = 3 days; the six-hour event consumes (6 ÷ 720) × 10 = 8.3% of the budget. Reasoning: 10× is below a one-hour 14.4× page threshold but above a six-hour 6× threshold, so the long-window alert fires while the short window confirms the burn is ongoing. That is the intended outcome - an 8%-of-budget event should not wake anyone up.
Traps
- Averaging latency percentiles across instances; percentiles are not additive - aggregate histograms instead.
- High-cardinality labels (user ID, request ID) on metrics, causing cost blowups and slow queries.
- Alerts that fire on every deploy; scope them with
for:durations or deploy suppression. - Sampling traces at 1% and then claiming you can investigate rare failures.