The Alert Storm Nobody Owned
A platform team of six supports 30 services. They have 412 alert rules, of which 280 fire at least once a week. The on-call rotation receives about 90 pages per week; responders estimate 70 are not actionable. Mean time to acknowledge is 4 minutes, but mean time to resolve is 6 hours because responders often cannot tell which service owns an alert. Two incidents in the last quarter were missed because their alerts were buried in the noise. The team has no error budget policy and no post-incident review process; incidents are closed when the page stops. Leadership asks for a dashboard showing alert volume by team.
1. Evaluate the request for an alert-volume dashboard. What is the more useful framing, and what would you measure first?
Read the worked answer
An alert-volume dashboard by team measures the symptom and risks turning noise into a competition. The more useful framing is: how much of the paging load is actionable, and how quickly can a responder identify an owner and act? Volume alone does not tell you whether the system is healthy; 90 pages a week could be fine if all are actionable, and 10 could be terrible if none are.
Measure first:
- Actionability rate: for each page, did the responder take a distinct action? With ~70 of 90 estimated non-actionable, the actionable rate is roughly 22%, which is the headline number. Target the inverse: at least 80% actionable.
- Ownership resolution time: time from page to identifying the owning service. The 6-hour MTTR with a 4-minute MTA suggests almost all the time is spent orienting, not fixing. Instrument this directly.
- Missed incidents: count of real incidents whose alert was not acted on in time. Two per quarter is the outcome metric that matters.
- Pages per rotation and per responder: to catch burnout and unfair distribution.
Then act: delete or downgrade the 280 weekly-firing rules that are not actionable, attach ownership metadata (team, runbook, dashboard link) to every remaining alert, and route by service rather than to a single firehose. The dashboard should show actionable rate and ownership time, not raw volume.
2. Design an incident response process that would have made the two missed incidents less likely, using the constraints of a six-person team.
Read the worked answer
With six people supporting 30 services, the process must be lightweight or it will be ignored.
1. Severity and routing. Define two or three severities. Only page-worthy conditions page a human; everything else is a ticket or dashboard. Route pages to the owning service's rotation, not one shared channel. This directly attacks the 'buried in noise' failure.
2. Every alert carries context. Each alert must include owner, runbook link, relevant dashboard, and a one-line description of what to do. An alert without an owner is a defect and is removed or fixed within a sprint.
3. Explicit incident command. For anything above a threshold, designate one responder as incident lead whose job is coordination and communication, not debugging. With six people, this prevents everyone diving into logs while nobody tracks the clock.
4. Error budget policy. Tie alert thresholds and release velocity to an agreed budget. When the budget is exhausted, releases pause until reliability work lands. This gives the team a defensible reason to prioritize noise reduction over features.
5. Blameless post-incident review. Within a week, a 30-minute review with a written timeline, contributing factors, and owned action items with dates. The two missed incidents should each get one; the question is why the alert was not actionable, not who missed it.
6. Track outcomes. Actionable rate, ownership time, missed-incident count, and percentage of reviews completed. Review these monthly and prune alerts continuously rather than in one heroic cleanup.