Alert Fatigue in a 120-Host Environment
A SaaS operations team monitors 120 hosts and three customer-facing services. The alerting pipeline generates about 420 alerts per day; 68% are false positives or duplicates. Mean time to repair has risen from 45 to 90 minutes over six months. Current paging alerts include CPU above 80% for five minutes, disk above 90%, HTTP 500 rate above 1%, and a synthetic check every 10 seconds. Six engineers share a weekly on-call rotation. In the last incident, a memory leak caused 500 errors for 22 minutes before anyone was paged, because the only related alert was CPU utilization, which stayed below threshold until the service was nearly down. The team wants to cut noise without missing real customer impact. They have service-level objectives for availability (99.9%) and latency (p95 under 800 ms), but alerts are not tied to them.
1. Redesign the paging alerts around symptoms and SLOs rather than raw causes. Explain which existing alerts you would keep, demote, or remove.
Read the worked answer
Page on customer-impact symptoms and SLO burn, not on every cause. Keep a multi-window error-budget burn alert for the 99.9% availability SLO: page when the 5-minute burn rate is high and confirm with a 1-hour window. Keep a p95 latency alert above 800 ms for 5 minutes if it correlates with customer impact. Keep the synthetic check but make it a multi-step transaction that asserts login and a core action; deduplicate it so one failure does not generate per-host alerts.
Demote CPU above 80% and disk above 90% to tickets or dashboards unless they are predictive of imminent outage. Those are causes, not symptoms, and they drive much of the noise. Remove duplicate per-host alerts in favor of service-level aggregation. Route each page to the owning service with a runbook link, and suppress alerts during approved maintenance. A page should mean a human must act now; everything else belongs in a queue.
2. Analyze why the memory leak was missed and propose detection changes that would have paged earlier.
Read the worked answer
The leak was missed because the only related signal was CPU, which can stay low while memory grows. The existing 500-rate alert either was not routed, was deduplicated, or did not fire at the right threshold; that must be verified. The synthetic check may have been too shallow to exercise the leaking path.
Add a memory working-set alert based on growth rate, not just absolute usage: for example, page when memory grows more than 10% per hour for 30 minutes and is above 80% of the limit, or when the container approaches OOM. Add a 500-rate alert at 1% for 2 minutes routed directly to the on-call, and a synthetic transaction that exercises the affected workflow end to end. Correlate memory growth with latency and error rate so a leak that degrades service pages before the process dies. Keep CPU as a diagnostic panel, not the primary detector.
3. Define a weekly review process and metrics to prove alert fatigue is decreasing.
Read the worked answer
Hold a 30-minute weekly review of the top 20 noisiest alerts and every page from the prior week. For each, classify it as actionable, duplicate, false positive, or missing coverage, assign an owner, and set a tuning or automation action with a due date. Review incident timelines to confirm detection time and whether the page led to a useful action.
Track alerts per week, percentage actionable, false-positive rate, pages per engineer per week, mean time to detect, mean time to repair, and the number of SLO-burn incidents. Set targets such as a 50% reduction in pages within four weeks, actionable ratio above 80%, and no increase in missed SLOs. Report the trend to engineering leadership. The proof is not fewer alerts alone; it is fewer alerts with stable or better MTTD and MTTR.