The 85% Burn in Nine Days
Checkout API publishes a 99.95% availability SLO over a rolling 28-day window, which allows about 20 minutes of unsuccessful requests per window. On day 9 of the current window, 85% of the budget is gone. The cause is a shared Redis cluster: the recommendations team runs periodic cache-flush experiments that briefly stall checkout writes. Two product launches are scheduled in the next week, and the VP of Product asks you to "just raise the SLO to 99.99 for the launch period" so the dashboard stops going red. Checkout has 14 downstream dependencies; two of them have no SLO at all.
1. The VP's request: what actually happens if you raise the SLO to 99.99% for the launch window, and what should you do instead?
Read the worked answer
Raising the target changes the measurement, not the system.
- 99.95% over 28 days = 0.05% x 40,320 min = ~20 minutes of budget.
- 99.99% over 28 days = 0.0001 x 40,320 min = ~4 minutes of budget.
Checkout has already burned ~17 minutes. At 99.99% you would be instantly and permanently out of budget, which means the SLO stops being a decision-making tool and becomes a number that is always red. Teams learn to ignore red numbers.
An SLO is a description of what users actually experience, not an aspiration or a marketing claim. Restating it does not add a single nine of real reliability.
What to do instead:
- Say plainly: "The SLO is currently accurate. The system is failing it." Keep 99.95%.
- Convert the request into a cost/benefit decision: to legitimately hold 99.99% you need isolation from the shared Redis cluster (dedicated cluster or write-path degradation), which costs roughly X engineer-weeks and Y/month in infrastructure. Give the VP that number and the date it could be delivered.
- Document the launch risk acceptance explicitly, with the VP as the owner, rather than laundering it through the SLO.
Revising a threshold while it is burning is the classic failure mode an error budget policy exists to prevent.
2. Error budget policy: with 85% burned and 19 days left, which actions are legitimate and which are overreaction?
Read the worked answer
First, do the arithmetic before you do anything dramatic.
- Burn so far: ~17.1 min in 9 days = ~1.9 min/day.
- Sustainable rate: 20.16 min / 28 days = ~0.72 min/day.
- That is a ~2.6x burn rate; remaining budget (~3 min) exhausts in roughly 1.5-2 days if the flush cadence continues.
Second, attribute the burn. This is the key judgement: the burn is caused by a dependency's behavior, not by checkout's own change velocity. The standard "freeze non-essential releases" consequence is the wrong lever here - stopping checkout deploys would not reduce the burn by one minute, and would force the team to break the policy for no benefit, teaching everyone the policy is theatre.
Legitimate actions:
- Raise the dependency issue to the recommendations team's leadership with the burn math attached; request a change freeze on cache-flush experiments during the launch window (a two-week, reversible, low-cost ask).
- Ship a degradation mode: if Redis write latency exceeds a threshold, queue writes and serve joins from the primary store.
- If neither lands before launch, get a written, named risk acceptance from the VP of Product for launching at 99.95%-minus. Unbudgeted risk is what the policy exists to make visible.
Overreaction: a blanket company-wide release freeze, or an emergency SLO rewrite. Neither touches the cause.
3. What is wrong with the current SLI design, and how would you extend coverage to the two dependencies with no SLO?
Read the worked answer
The headline SLI is right; the attribution layer is missing.
- Measuring success rate at the edge (join requests that succeed from the client's point of view) is correct - it captures user pain regardless of which internal component is at fault. Keep it.
- But it cannot tell you where the failure came from. When checkout goes red, responders will start at the checkout service and spend the first 20 minutes of a 45-minute incident looking in the wrong place.
Add per-dependency SLIs at the call site, not inside the dependency. You cannot force another team to instrument on your schedule, and their definition of "healthy" may differ from yours. Measure what your service observes: Redis operation error rate, p99 write latency, and abort/timeout rate from checkout's own client library. Those numbers are yours, they are defensible, and they make attribution a 10-second glance.
For the two dependencies with no SLO:
- Don't demand they adopt SLOs - that's a multi-month organizational negotiation for a 14-dependency service.
- Instead define a dependency budget: an internal, unglamorous threshold (e.g., "calls to service X must succeed 99.99% of the time for checkout to meet 99.95%"). Derive it from the math: a single dependency contributing more than ~10% of the total budget is a structural risk.
- Only escalate to a formal SLO conversation for dependencies that breach that threshold.