Alert Flood During a Traffic Event: Mute or Redesign?
A B2B SaaS runs 280 services and 210 alert policies across 60 on-call engineers. During a two-hour marketing event, traffic doubles and 1,180 notifications arrive; 85% come from six static policies such as CPU above 80% for five minutes and error rate above 1%. Incident correlation groups the notifications into three incidents, but a real payment-provider outage is not included because the payment policy is muted after a previous noisy deploy. The team proposes muting the six noisy policies and raising thresholds. SLO burn-rate alerts exist but were created last quarter and have not been reviewed. Mean time to acknowledge is 14 minutes; mean time to detect is 9 minutes.
1. Evaluate the proposal to mute the six policies and raise thresholds.
Read the worked answer
The proposal treats symptoms as the problem. Muting removes the safety net entirely, and the payment outage shows why that is dangerous: a mute without an owner and expiry outlived the condition it was meant to suppress. Raising static thresholds may reduce pages during normal traffic, but it also delays detection when a real saturation event occurs, because the new threshold is arbitrary rather than tied to user impact.
The CPU policy is especially weak during a traffic event. Doubling traffic should raise CPU; paging on CPU above 80% without a baseline confuses expected load with failure. A better question is whether latency, errors, or queue depth are degrading. Error rate above 1% is also context-free: a 1% error rate on a low-volume endpoint may be noise, while a smaller error rate on payment may be severe.
The team should not mute first and investigate later. It should classify the six policies, identify which represent user-impacting symptoms, and replace cause-based static thresholds with service-level and anomaly-based conditions. Any temporary mute needs an expiry, a reason, and an owner.
2. Design a tiered alerting and correlation approach for the next traffic event.
Read the worked answer
Use three tiers:
- Customer-impact pages: multi-window SLO burn-rate alerts for each critical user journey, routed to the owning service. These are the primary pages.
- Early-warning and capacity alerts: anomaly detection or dynamic baselines for CPU, memory, queue depth, and error rate. These create tickets or low-urgency notifications, not pages, unless combined with a symptom.
- Diagnostic signals: dashboards and service-map views for responders, not individual pages.
For correlation to work, every signal needs consistent entity tags, service names, and environment labels. The payment policy should be re-enabled with a time-bound mute and tested. Alert grouping should merge symptoms from one dependency, but it must not hide a distinct customer journey; retain at least one independent payment synthetic or SLO alert.
Use AI-assisted correlation as an aid, not an autopilot. It can group and rank, but responders still need a runbook and an accountable owner. Review the SLO alerts quarterly so they reflect current traffic and dependencies.
3. How would you validate that the redesign actually improves detection without hiding incidents?
Read the worked answer
Backtest against historical alerts from the traffic event and from at least one normal week. Measure precision and recall: how many pages were truly customer-impacting, how many noisy pages were removed, and whether any real incident such as the payment outage would have been missed. Replay the payment scenario with the new rules and confirm it pages independently of the grouped incident.
Run a game day that doubles load in a controlled environment and injects a dependency failure. Track mean time to detect, mean time to acknowledge, and the number of notifications per responder. Compare with the 9-minute and 14-minute baselines. Set a target for pages per shift and review false negatives as seriously as false positives.
Finally, make the review operational: a weekly alert-quality review, an expiry date on every mute, and a rule that no policy is muted without an owner and a linked incident. The validation is not a one-time exercise; it is evidence that the new system detects the payment outage and reduces noise at the same time.