On-Call Alert Fatigue After a Migration
A team migrated 38 services from VMs to a managed container platform over three months. Since the migration, the on-call rotation receives about 140 alerts per week, up from 20. Roughly 70% are closed as no-action. The most frequent alert is CPU above 80% for 5 minutes, which fires on batch jobs that are expected to spike. Page-worthy incidents are now sometimes acknowledged late because responders triage noise first. Mean time to acknowledge has risen from 4 minutes to 19 minutes. The team has 6 engineers in the rotation, and two have asked to leave on-call. The platform team says the alert thresholds were copied from the VM-era monitoring configuration.
1. Why is the migration correlated with the alert increase, and what does the MTTA change indicate?
Read the worked answer
The migration changed the resource model, but the monitoring was carried over unchanged. On VMs, CPU above 80% for 5 minutes often meant a host was saturated and a workload was at risk. On a managed container platform, CPU is a request-and-limit model: a container can sit at 80% of its limit while the node has ample capacity, and batch jobs are designed to spike. The threshold no longer measures what it used to. The same applies to memory and to any host-level checks that were translated one-to-one into container metrics.
The 140 versus 20 alerts per week and the 70% no-action rate confirm that most of the new signal is not actionable. That is the definition of alert fatigue.
The MTTA increase from 4 to 19 minutes is the more serious indicator. It is not that responders are slower; it is that they must triage a noisy queue before they can tell which alert is real. When 70% of pages are false, responders learn to wait and see, and that learned delay applies to the 30% that matter. The two engineers asking to leave on-call is the predictable end state: sustained false paging erodes trust in the system and pushes people out of the rotation.
The root cause is a monitoring configuration that was not re-derived for the new platform, not a sudden increase in real incidents.
2. Design a remediation plan with measurable targets. What should the team change first?
Read the worked answer
First, stop the bleeding on the noisiest signal. Take the CPU-above-80% alert and either delete it or convert it to a dashboard-only metric. It accounts for a large share of the 140 weekly alerts and is almost never actionable in the container model. Replace it with alerts on symptoms users feel: error rate, latency against SLO, saturation of the node pool, and pod restart loops.
Second, classify every remaining alert as page, ticket, or dashboard. Anything that has been closed as no-action more than twice in a month does not belong in the paging path. Add a required field to the alert-close process so the no-action rate is measured, not estimated.
Third, set targets and track them weekly:
- Weekly pages reduced from 140 to under 30 within 30 days.
- No-action rate below 10%.
- MTTA back under 5 minutes for page-worthy alerts.
- At least one alert reviewed per engineer per week, with a named owner for each alert rule.
Fourth, fix the process that let this happen. Alert rules should be reviewed when a workload moves platforms, and the review should include the on-call responders, not only the platform team. Add a monthly alert review to the team's operating rhythm.
Finally, address the people risk. Two engineers want out of the rotation; reducing noise is the retention lever. Until pages drop, consider a temporary secondary rotation so no one is alone in the queue. Measure the result and report it, because the team will not believe the fix until the page volume actually falls.