Error Budget Under Pressure
Atlas SaaS provides a B2B billing API. The availability SLO is 99.95% monthly, which yields about 21.6 minutes of allowed downtime. Operations tracks only 'server up' per host and reports 99.99% uptime; last month a database failover caused 43 minutes of API errors, but the dashboard showed green because the hosts were running. The team has 400 alerts per week, and 72% are false positives or duplicates; mean time to restore is four hours. Product management wants weekly releases and argues that the error budget is 'an excuse for missing deadlines.' A recent incident review found that the failover runbook was 14 months old and no one had tested it. The SRE team proposes burning the error budget on feature work if the budget is healthy, but no policy exists for what happens when it is exhausted. The on-call rotation is five engineers, two of whom handle most escalations.
1. Critique the current availability measurement and define a better SLI and SLO for the billing API.
Read the worked answer
Host uptime is not a user-facing reliability measure. Last month the dashboard was green while the API failed for 43 minutes because the hosts were running. The SLI should be the proportion of valid API requests that succeed within a latency threshold, measured at the load balancer or gateway, not on individual hosts. For example: successful responses with status below 500 and latency under 500 ms divided by all valid requests, measured over a rolling 28-day window. The SLO might remain 99.95% availability, but it should be expressed in terms of that request-based SLI. A 43-minute outage would then consume the monthly error budget of about 21.6 minutes and trigger the policy. Add a second SLI for latency if billing calls have a user-visible timeout. Define the measurement window and exclusions in advance, such as planned maintenance, and publish the calculation. The current 99.99% host uptime is a vanity metric that hides user pain and prevents learning.
2. What error-budget policy should Atlas adopt when the budget is healthy and when it is exhausted? How should product management be involved?
Read the worked answer
Agree the policy before the next incident, with product, SRE, and engineering leadership. When the budget is healthy, feature releases can proceed at the agreed cadence, and reliability work is funded from a standing allocation. When the budget is exhausted, freeze non-essential feature releases and prioritize reliability work until the service is back within SLO for an agreed period, for example two consecutive weeks. Exceptions should require an executive risk decision, not a quiet override. Product management should co-own the policy because it trades feature risk for reliability risk. The policy is not an excuse for missing deadlines; it is a pre-agreed rule that prevents an outage from becoming a blame debate. Also define burn-rate alerts: a fast burn over a short window should page, while a slow burn creates a ticket. Publish the remaining budget weekly. If product argues the SLO is too strict, revisit the SLO with customer evidence, not in the middle of an incident.
3. What should Atlas fix first to reduce alert noise and improve resilience? Explain the sequence.
Read the worked answer
First, reduce alert noise because 400 alerts per week with 72% false positives trains responders to ignore pages. Deduplicate, remove host-level alerts that do not map to user symptoms, and replace them with symptom-based alerts and SLO burn-rate alerts. Route only actionable pages to the on-call rotation; send the rest to a queue. Second, test the failover runbook and update it. A 14-month-old, untested runbook explains part of the four-hour restore time. Run a game day for the database failover, record what breaks, and automate the steps that can be automated. Third, address on-call concentration: two of five engineers handle most escalations, which is a resilience risk. Pair them with others on real incidents and rotate. Sequence matters because better alerting gives responders attention for the game day, and the game day produces evidence for automation. Do not start with a large chaos program; start with one known failure mode and measure time to detect and restore.