Error Budget Burn on a Checkout API
Your team runs a checkout API on IBM Cloud Kubernetes Service across three availability zones in Dallas. The service-level objective is 99.9% availability over a rolling 30-day window, which allows about 43 minutes of unavailability. On day 3 of the window, a release changes a readiness probe path. For 22 minutes all pods are removed from the load balancer and every checkout request fails. After the probe is fixed, the API runs for another 90 minutes at an 8% error rate before a configuration rollback restores normal behavior. The product team wants to keep a planned release on day 5 because it is already announced to customers. Your error-budget policy says that if more than 50% of the budget is consumed in the first 10% of the window, non-emergency releases must stop. The on-call engineer is unsure whether the incident is severe enough to trigger the policy and asks you to decide.
1. Calculate the error budget consumed by the incident, including the degraded period, and decide whether the day-5 release should proceed. Show your reasoning and state any assumptions.
Read the worked answer
A 99.9% SLO over 30 days allows 0.1% of 43,200 minutes, or 43.2 minutes of bad time. The total outage is 22 minutes. The degraded period adds 90 x 0.08 = 7.2 equivalent minutes. Total consumption is 29.2 minutes, or 29.2 / 43.2 = 67.6% of the budget. That is more than the 50% threshold, and day 3 is inside the first 10% of the window (3 days out of 30), so the policy triggers. The day-5 release should not proceed as a normal release. Assumptions: every failed request counts equally, the 8% error rate is measured at the load balancer rather than only at pods, and no other budget was consumed earlier. The correct action is to halt non-emergency changes, require an exception approved by the service owner and incident commander, and only allow the release if it includes a canary, a verified rollback, and a documented reason that waiting would create greater customer risk.
2. The current alert is a static threshold: page when the 5xx rate exceeds 2% for 5 minutes. Explain why this alert is a poor fit and describe a multiwindow burn-rate alerting scheme for this SLO.
Read the worked answer
A fixed 2% threshold is not tied to the 43.2-minute budget. On a low-traffic service it can page for a handful of failures, while a sustained 1.5% error rate can exhaust the budget without ever firing. It also gives no sense of urgency and tends to produce repeated pages during one incident. A burn-rate scheme compares the observed error rate with the rate the budget allows. For a 99.9% SLO, budget spend of 1x means consuming 43.2 minutes over 30 days. Page on a fast burn: 14.4x over both 1 hour and 5 minutes. Page on a medium burn: 6x over both 6 hours and 30 minutes. Create a ticket, not a page, for 1x over 3 days. Use the error ratio from the ingress or load balancer and the application, require both windows to agree, and attach a runbook that tells the on-call engineer whether to roll back, scale, or fail over.
3. What changes would you make to the deployment process and SLO review to prevent a readiness-probe change from causing a full outage?
Read the worked answer
The failure came from a configuration change that removed every pod from service, so the control must be both preventive and detective. Preventive: make the readiness probe part of a tested manifest, validate it in staging with traffic, set maxUnavailable and a pod disruption budget, and add a canary stage that receives 5% of traffic before full rollout. Do not let liveness and readiness share the same endpoint or timing assumptions. Detective: add an automated gate that compares canary error rate and latency against the baseline and rolls back automatically. Also add a deploy-time check that the new pod becomes Ready before old pods are terminated. For SLO review, review burn rate weekly, not only monthly, and record every change that consumes budget. The day-5 decision should be an explicit error-budget exception, not an informal judgement, so the same failure mode does not get repeated for schedule reasons.