Alert Fatigue on a 200-Server Estate
Adventure Works monitors 200 Windows Server 2022 hosts with a mix of Performance Monitor data collector sets, Event Viewer subscriptions, and a third-party agent. Operations receives ~1,400 alerts per day; 85% are dismissed without action. The most common alert is "CPU above 80% for 5 minutes" on the 20 SQL Server hosts, which spikes during nightly index maintenance and is expected. Meanwhile, a disk-full incident on a file server went unnoticed for two days because the alert was buried. The team wants to reduce noise without missing real incidents, and asks whether to keep the third-party agent or standardize on native tooling.
1. How should the CPU alert be tuned so it stops firing on expected maintenance but still catches genuine saturation?
Read the worked answer
Replace the static threshold with a context-aware rule. A flat "CPU > 80% for 5 minutes" cannot distinguish index maintenance from a runaway query. Options, in order of preference:
- Suppress during a known window. On the 20 SQL hosts, suppress or downgrade the CPU alert during the scheduled index-maintenance window (e.g., 01:00–03:00), since the spike is expected. This is the simplest fix and removes most of the noise.
- Add a second condition. Alert only when CPU > 80% and the SQL Server process is not the top consumer, or when CPU > 80% persists outside the window for more than 15 minutes. This catches a non-SQL runaway process.
- Use a baseline/relative threshold. Alert when CPU exceeds the host's normal 95th-percentile baseline by a meaningful margin, rather than a fixed number.
Also fix the root alert design. 1,400 alerts/day with 85% dismissal means the alert set is not actionable. Classify each rule as page / ticket / log, and delete or downgrade rules that never produce action. The disk-full miss shows the opposite failure: the rule existed but had no severity or routing that made it visible - so disk-full should be a high-severity page, not a low-priority entry in a flood.
2. How should the disk-full miss be prevented, and what native capability covers it if the third-party agent is removed?
Read the worked answer
Prevent the miss with severity and routing, not more alerts. The disk-full rule should be high severity, routed to the on-call queue (not a general mailbox), and have a low threshold with a warning tier (e.g., 80% warning, 90% critical). Test the route quarterly by injecting a synthetic alert.
Native coverage. Windows Server 2022 provides:
- Performance Monitor data collector sets with an alert on
LogicalDisk(_Total)\% Free Space, which can run a task or write to the event log. - Event Viewer subscriptions to forward specific events (e.g., disk warning events) to a central collector.
- Storage Sense / storage reports for capacity trending, and Azure Monitor / Log Analytics if the estate is hybrid, which gives alerting, routing, and dashboards.
Gap to acknowledge. Native tooling is weaker at centralized routing, suppression, and on-call escalation than a mature third-party agent. If the team removes the agent, it must build routing (e.g., Log Analytics alert rules with action groups) or accept a manual process. The decision should be based on whether the team can operate that routing reliably - not on tool preference.
3. The team proposes deleting all alerts that were dismissed more than 100 times. Evaluate this approach.
Read the worked answer
Reject it as stated - dismissal count is not a measure of value. A rule dismissed 100 times may be a real signal that is mis-thresholded (like the CPU rule) or routed to the wrong person; deleting it removes detection entirely. Conversely, a rule dismissed 100 times may genuinely be noise. The count alone cannot tell you which.
Better process. For each high-dismissal rule, ask: (a) Did any dismissal correspond to a real incident? (b) Is the threshold or window wrong? (c) Is it routed to someone who cannot act? Then tune, re-route, or retire - and record the decision. Retire only rules that have produced zero actionable incidents over a defined period (e.g., 90 days) and whose underlying condition is covered elsewhere.
Guardrail. Before retiring, confirm the failure mode is still detected by another control (e.g., capacity trending, a lower-severity warning). The disk-full miss is the cautionary tale: the alert existed, but the system around it failed. Deleting rules by dismissal count would have removed it entirely.