The SLO that stayed green through 41 minutes of downtime
TicketNova sells concert tickets. The purchase path is a Cloud Run front end, a GKE checkout service, and an HA Cloud SQL primary. Product promised 99.95% availability measured across each 90-minute on-sale window, when 40,000 concurrent users arrive; the rest of the month traffic is near zero. Last month the path was fully unavailable for 41 minutes in three incidents: 22 minutes of connection-pool exhaustion after a schema migration, 12 minutes from a bad feature flag, and 7 minutes after a zonal Cloud SQL failover. The single SLO dashboard shows a rolling 30-day 99.9% availability target measured in minutes of full outage. No alerts fire from it. The weekly release train has never been paused. Leadership asks why the SLO did not catch the outage.
1. Did the SLO measurement fail, the target fail, or the process fail? Show the arithmetic.
Read the worked answer
Run the numbers first. Thirty days is 43,200 minutes, so 99.9% buys a 43.2-minute error budget. The 41 minutes consumed 95% of it, but availability still read 99.905% - above target. The dashboard was arithmetically correct and operationally useless.
Three separate errors:
- Measurement. Minutes-of-full-outage ignores partial degradation. A 12-minute window at 40% error is 7.2 minutes of budget under a request-success ratio, not 12. More importantly, a ratio weights on-sale traffic (8x) properly; a time-based metric treats an on-sale Tuesday evening and a quiet Sunday dawn as equally valuable.
- Target shape. The business cares about 90-minute windows. Losing 41 minutes inside one on-sale is a lost evening, not 0.1% of a month. A monthly target can never express that.
- Process. Nothing consumed the budget. No alert, no freeze, so 95% burn had no consequence.
Replace with two SLOs: purchase-path good-request ratio at 99.9% over a rolling 28 days as the release-blocking number, plus a per-on-sale-window SLO at 99.95% good requests with its own budget. The 7-minute zonal failover also says the client does not retry - an HA failover should be seconds.
2. Design the alerting on these SLOs. What pages a human at 3 a.m., and what does not?
Read the worked answer
Alert on burn rate against the budget, multi-window, and page only on the fast burns.
With a 43.2-minute budget over 28 days:
| Rule | Condition | Budget consumed | Action |
|---|---|---|---|
| Fast burn | 14.4x over 1h, confirmed by 5m | ~2% in one hour | page |
| Medium burn | 6x over 6h, confirmed by 30m | ~5% in six hours | page |
| Slow burn | 1x over 3d, confirmed by 6h | ~10% in three days | ticket |
The short confirmation window is what keeps the 1-hour rule from paging on a 90-second blip; requiring both windows is the whole point.
For the on-sale SLO, add a separate rule that only evaluates during a sale window and pages at a much faster rate, because the entire budget is 90 minutes long.
Do not page on CPU, memory, or pod restarts - those are causes, not symptoms. Page on the good-request ratio and on client-observed latency, both of which are what the customer feels. Also alert on missing data (synthetic probe not reporting) so a silently broken dashboard does not look like a healthy system, which is exactly how this team learned about the incident.
3. Finance wants to advertise 99.95% end-to-end. The payments provider publishes 99.95%. Is that defensible, and what would you do instead?
Read the worked answer
No. Availability composes multiplicatively when you cannot serve without the dependency: 0.9995 x 0.9995 = 0.99900. Advertising 99.95% means you have just promised the customer your provider's number as if it were free, and any failure on your own side breaks the promise.
The SRE discipline is budget allocation. If the purchase path carries 99.95% (about 21.6 minutes over 28 days), decide deliberately how much of that budget the provider is allowed to spend - commonly 10%. That means the provider must sustain roughly 99.995% during your windows, which is a hard conversation, or you absorb the gap with engineering:
- A second PSP with health-checked weighted routing, so provider failures become a degraded path rather than an outage.
- Queue and retry authorization asynchronously with idempotency keys, converting a hard failure into a slow success. This matters because a request served in 8 seconds still counts as good if your SLO has a generous latency bound - a request served never does not.
- Circuit breaking so the checkout service fails fast instead of exhausting its thread pool while the provider hangs.
If none of that is affordable, the honest move is to advertise 99.9% and keep the 99.95% internal. A published number that the architecture cannot support is a future outage review that is already written.