Apply Site Reliability Engineering Practices
From SLI to error budget
An SLI is a measured ratio of good events to valid events: availability, latency under a threshold, freshness, correctness. An SLO is the target for that SLI over a window, such as 99.9% of requests succeeding over 28 days. An SLA is a contract with financial consequences and should be looser than the internal SLO. Never target 100%: users cannot distinguish 99.99% from 100%, but the cost curve goes vertical.
The error budget is 1 − SLO. For 99.9% over 30 days that is about 43 minutes. Budget consumption is expressed as a burn rate: a burn rate of 1 exhausts the whole budget exactly over the window. A 30-day window is 720 hours, so burning 2% of the budget in one hour is 0.02 × 720 ÷ 1 = 14.4× burn rate. The recommended alerting pattern is multiwindow, multi-burn-rate: a fast burn (14.4× over 1h, confirmed by a 5-minute short window) pages a human, while a slow burn (for example 6× over 6h) opens a ticket. The short confirmation window prevents flapping.
Also expect questions on toil reduction, blameless postmortems with owned action items, capacity planning driven by SLO headroom, progressive delivery (canary with automated analysis and rollback on SLO breach), and GameDay or chaos exercises. Distinguish monitoring, which covers known failure modes via dashboards and alerts, from observability, which covers novel failure modes via high-cardinality traces and structured logs.
Worked example: a 99.95% SLO over 28 days gives a budget of 0.0005 × 40,320 minutes ≈ 20 minutes. A 4-minute outage consumed 20% of the budget; four equivalent incidents would exhaust it and should trigger a release freeze and reliability work.
Rule of thumb: alert on symptoms users feel (SLO burn), not on every underlying cause metric.