SRE Practitioner study materials
SRE Practitioner study resources

The Error Budget Is Empty and the Launch Is Committed

SRE Practitioner · Put your knowledge to work in context.

All case studies

The Error Budget Is Empty and the Launch Is Committed

A retailer's checkout platform handles 12k rps at peak. The user-facing SLO is 99.9% of requests served in under 400 ms over a 28-day rolling window, measured at the edge load balancer. On day 11 of the window the error budget is 96% consumed: a database failover burned about 20 minutes of total unavailability and a bad feature-flag push burned about 18 minutes. Availability, tracked as a separate SLI, still reads 99.97%. In nine days the payments team must ship a new card processor to hit a marketing commitment worth roughly $400k. The written error-budget policy says "feature freeze when the budget is exhausted", but it predates the availability SLI and has never been invoked.

1. The VP of Product asks you to approve the launch. What is your recommendation, and what evidence do you put on the table first?

Read the worked answer

Recommendation: do not answer yes or no yet - reconcile the budget math, then decide.

  1. Size the pool. 0.1% of 28 days is ~40 minutes of full-outage-equivalent. A 20-minute total outage is roughly half the budget, not a rounding error. Total outages are charged at ~100% request failure, not at the fraction of time anyone noticed.
  2. Ask which SLI actually burned. Requests that fail outright are an availability event, not a latency event. Two classic accounting errors: excluding failed requests from the latency SLI (which hides outages inside a 'healthy' latency number), and measuring at the edge LB where retries mask origin slowness. A 99.97% availability figure next to a 96%-consumed latency budget almost certainly means the two SLIs are counting different request populations. Reconcile before quoting either.
  3. Separate symptom from cause. The bad flag push is self-inflicted and is exactly what the budget exists to prevent. The DB failover is an infrastructure bet; if it recurs, it is a reliability-work item, not a launch veto.
  4. Bring the burn rate, not just the total. At 96% consumed by day 11 you are burning roughly 8x the sustainable rate; a multiwindow burn alert (1h/5m and 6h/30m) should already be firing.

The decision question is not 'is the number over 100%' but 'what new risk does this change add to a pool with no headroom left'.

2. The policy says freeze, but the policy has never been enforced and the org wants to ship. How do you handle the exception request?

Read the worked answer

Treat it as a formal, time-boxed exception with a named risk owner - not a quiet override, and not a heroic refusal either.

  • Cost it in customer terms, then hand the decision to the person who owns the consequence. Written risk acceptance by a named exec: what could break, who owns the customer impact, what the rollback is.
  • Buy down the risk instead of arguing about the freeze. Canary 1% → 5% → 25% with automatic rollback on a burn-rate alert; the new provider behind a traffic split so it can be disabled in under a minute; a flag kill switch; doubled on-call for launch week; a pre-agreed abort threshold such as 'any 5xx increase above 0.05% sustained for 5 minutes → revert'.
  • Scope the freeze to what it protects. The policy protects the checkout SLO. Freezing unrelated services is collateral damage, and collateral damage is how a policy gets repealed.
  • Fix the policy afterwards. An untested freeze clause is governance theatre. Add multiwindow burn thresholds, who may grant exceptions and for how long, and what happens if the budget is exhausted two windows in a row.

The anti-pattern to avoid is negotiating the budget number itself. If exceptions become routine, the honest message is that 99.9% is not resourced: either fund the reliability work or formally lower the SLO and accept the risk.

3. Should this launch be gated on the latency SLO or the availability SLO? Explain how you reason about that.

Read the worked answer

Gate on the SLO the change can actually degrade, while being explicit that an error budget is a per-service risk pool, not a per-change ledger.

  • The payment-provider change alters the error/decline path, so it puts availability and correctness at risk. Correctness is usually better guarded by its own SLI (authorization success rate, chargeback rate) than by latency.
  • The DB failover burn was latency/availability damage from an infrastructure event. It says nothing about this launch's risk profile - but it does say the service has no headroom to absorb a new failure mode.
  • Do not split the budget into fake per-change accounts. That is error-budget theatre and it invites gaming. Use the whole-service budget as the gate, and use the change's failure-mode analysis to set the guardrails. If the launch must proceed during an exhausted window, the safeguards have to be strong enough that a bad launch cannot compound existing debt.

If the org finds itself making this call every sprint, the diagnosis is not 'product is reckless' - it is that the reliability target and the reliability funding are mismatched. Say that in the postmortem of the policy, not in the incident channel.