Node CPU Alert Storm During a Batch Window
Infrastructure Monitoring covers 30 hosts. A nightly batch job runs 01:00-03:00 and legitimately drives one worker node's CPU to 95% for the duration. The team has a static CPU threshold alert at 85% on all nodes, which fires every night and pages the on-call engineer, who has started ignoring it. Separately, a real incident last month - a memory leak on a different node - was missed because the on-call was desensitized. Business Analytics shows order throughput is normal during the batch window. The team wants fewer pages without missing real problems.
1. Why is the current alert design failing, and what would you change about the CPU alert specifically?
Read the worked answer
A static 85% threshold on all nodes ignores baseline behavior. The worker node's 95% is its normal nightly pattern, so the alert has no signal - it fires on expected behavior and trains the on-call to ignore pages (alert fatigue), which is why the real memory leak was missed. The threshold is also uniform across heterogeneous nodes.
Change: make the CPU alert baseline-aware (dynamic/anomaly-based) so it fires on deviation from each node's normal pattern rather than an absolute number. For the worker node, either exclude it from the generic CPU policy or use a condition that recognizes the 01:00-03:00 batch window as expected. Keep a separate, higher absolute threshold as a safety net for genuinely abnormal saturation outside the window. Verify the change reduces nightly noise while still catching true anomalies.
2. How would you use Business Analytics to reduce false pages and prioritize real ones?
Read the worked answer
Business Analytics shows order throughput is normal during the batch window, which is the key context: high CPU with normal throughput is expected work, not an incident. Tie alert severity to business impact - if throughput and success rate are healthy, a CPU spike is informational, not a page. Conversely, if throughput drops or errors rise while CPU is high, that's a real page.
Practically: enrich the alerting logic so CPU alerts are suppressed or downgraded when business KPIs are normal, and escalate when business KPIs degrade. This uses the business signal to distinguish 'busy but fine' from 'busy and broken,' reducing noise without hiding impact.
3. Design the overall approach so the memory-leak class of incident is caught while pages drop.
Read the worked answer
Move from uniform static thresholds to per-node, per-metric baselines plus health rules that combine multiple signals. For memory, an absolute threshold is weak; a steadily rising memory trend with no release (leak signature) should alert even below a hard cap. Use health rules that evaluate rate-of-change and sustained deviation, and combine host metrics with business KPIs so severity reflects impact.
Tier the response: informational events for expected batch-window saturation, warnings for anomalies without business impact, pages only for anomalies with business impact or clear failure signatures (e.g., memory climbing monotonically toward exhaustion). Then validate against history: replay the leak period and the batch window to confirm the new rules would have caught the leak and stayed quiet on the batch. Fewer, higher-signal pages is the goal, not fewer detections.