Performance, Reliability, and Monitoring
Performance, Reliability, and Monitoring tests whether you can define what "working" means numerically and detect drift before users report it.
SLIs, SLOs, and error budgets
- An SLI is a measured ratio: proportion of requests served in under 300 ms without a 5xx, measured at the edge or client.
- An SLO is a target over a window: 99.9% over 30 days = 43.2 minutes of allowed failure (0.001 × 43,200 minutes).
- Error budget = 1 − SLO. Burn rate = observed error rate ÷ budget rate; a 10x burn exhausts a 30-day budget in 3 days. Pair a fast window (1h/5m) with a slow window (6h/30m) to catch spikes and slow bleeds.
Signals
- Golden signals: latency reported as p50/p95/p99 - never the mean - plus traffic, errors, and saturation.
- RED for request-driven services (rate, errors, duration); USE for resources (utilization, saturation, errors).
- Distributed tracing with propagated trace IDs, structured correlated logs, and cheap aggregated metrics.
Resilience patterns
Timeouts, bounded retries with jitter, circuit breakers, bulkheads, priority-based load shedding, and caching with explicit invalidation. Exercise them with load and chaos testing before production does it for you.
Worked example
Checkout SLO: 99.9% success over 30 days. One hour shows 1% errors. Budget consumed = 1% × (1/720) of the window ÷ 0.1% = 1.39% of the monthly budget, and burn rate = 1% / 0.1% = 10x, which would exhaust the month in 3 days. A fast page typically fires at 14.4x, so this sits below the page threshold but above the slow-burn ticket threshold: investigate now, do not page. The reasoning is the point - page on fast severe burn, ticket on slow burn.
Traps
- Averaging latency so a p99 regression hides inside the mean.
- Alerting on CPU instead of user-visible symptoms.
- Measuring availability server-side while clients fail at DNS or TLS.
- Retries that amplify load during an incident.
- SLOs with no agreed action when the budget is exhausted.