Alerting on Symptoms When the SLO Budget Is Small
A SaaS provider has a 30-day availability SLO of 99.9% for its public API, measured at the load balancer. The error budget is 43 minutes per 30 days. On-call rotation has two engineers, and the team receives about 180 alerts per week; 70% are closed without action. Last month, the API had a 6-minute outage caused by a bad config push. The alert that fired was a high CPU alert on one node, while the load balancer 5xx rate alert was configured with a 15-minute window and a 5% threshold. The post-incident review found that the 5xx rate reached 40% for 6 minutes but never stayed above 5% for 15 minutes. The team wants to reduce alert volume and detect future incidents faster. They propose deleting all CPU and memory alerts and relying only on a single 5xx alert with a 1-minute window and a 0.1% threshold.
1. Calculate the burn rate for the 6-minute incident against the 30-day SLO, and explain why the existing 5xx alert did not fire.
Read the worked answer
The 99.9% SLO allows 0.1% errors. Over 30 days, that is 43,200 minutes times 0.001, or 43.2 minutes of full outage equivalent. During the incident, 40% errors for 6 minutes consumed 6 times 0.40, or 2.4 error-minutes. That is 2.4 divided by 43.2, about 5.6% of the monthly error budget.
The instantaneous burn rate is 40% divided by 0.1%, or 400 times the sustainable rate. Over a one-hour window, the same 6 minutes at 40% produces a 4% error rate and a 40x burn rate. The existing alert used a 15-minute window and a 5% threshold, and the setup states the rate never stayed above 5% for 15 minutes. A 6-minute spike cannot satisfy a 15-minute continuous condition, so the alert suppressed exactly the kind of short, severe event that matters. The CPU alert fired because CPU rose on one node, but CPU is a cause-oriented signal, not a user-facing symptom.
The lesson is not that 5% is always wrong; it is that a long for duration can hide a fast burn. A multiwindow rule would have caught the 6-minute event while still suppressing brief blips.
2. Evaluate the proposal to delete all CPU and memory alerts and use a single 0.1% 1-minute 5xx alert. What alerting design would you recommend?
Read the worked answer
The proposal overcorrects. Deleting all resource alerts loses capacity and saturation signals that are useful for tickets and dashboards, even if they should not page. A single 5xx alert at 0.1% over 1 minute will also be noisy: on low-traffic periods one or two errors can exceed the threshold, and it has no long-window confirmation. It may page for harmless blips while still missing a slow burn.
Use symptom-based SLO alerting with multiple burn-rate windows. A common pattern is to page on a fast burn, such as 14.4x over 1 hour with a 5-minute short window, and on a slower burn, such as 6x over 6 hours with a 30-minute short window. Use a ticket-level alert for a 1x burn over 3 days with a 6-hour short window. Require both the long and short windows to be above threshold so a brief spike does not page by itself, but a sustained fast burn does. Add latency, saturation, and synthetic checks where they represent user impact.
Keep CPU and memory alerts as non-paging tickets or dashboard indicators, and page only when they correlate with an SLO symptom or a clear capacity risk. Use traffic-aware thresholds, deduplicate related alerts, and attach a runbook and ownership to every page. The goal is fewer pages, not fewer signals.
3. How should the team change its on-call and post-incident process to reduce non-actionable alerts?
Read the worked answer
Treat alert quality as an operational product. Require every page to have an owner, a runbook, a clear action, and a link to an SLO or user symptom. If an alert is closed without action repeatedly, either automate the action, convert it to a ticket, or delete it. Review the top alert sources weekly with the on-call engineers, not only after incidents.
After the bad config push, the blameless review should ask why the change was not caught by a canary or config validation, why the 5xx alert's duration hid the event, and why the CPU alert was the only page. Add a canary stage with automated rollback, a config validation step, and a fast-burn SLO alert. Define an error budget policy: when the budget is exhausted, releases require additional review until reliability work is completed.
With only two engineers on call, fatigue is a reliability risk. Cap pages, rotate follow-the-sun or escalate earlier, and protect sleep with non-urgent routing. Track pages per shift, actionability rate, time to acknowledge, and the percentage of incidents detected by SLO alerts rather than resource alerts.