Alert Noise and the Case for Anomaly Detection Everywhere
Halvard Logistics has run 210 hosts and 60 services on New Relic for six months. Their alert policy was copied from the previous tool: CPU above 80% for five minutes, error rate above 1%, and host memory above 90%, all evaluated per host. During a peak shipping window the on-call engineer received 638 notifications in one eight-hour shift. Seventy-one percent were CPU alerts on batch hosts whose normal operating band in that window is 85-95%; twelve notifications corresponded to genuine latency incidents. Leadership wants anomaly detection enabled everywhere and static thresholds muted. The SRE team is split: two services have strong weekly seasonality that a short baseline does not capture, and one real incident was a 40-minute slow ramp in checkout latency that never crossed a static threshold. One engineer is available for three weeks.
1. Evaluate the proposal to mute every static threshold in favour of anomaly detection. What should stay static and what should become baseline-driven?
Read the worked answer
Reject the blanket swap; use a split by failure mode.
Keep static thresholds where the condition is an absolute invariant rather than a statistical deviation: certificate and licence expiry, disk or volume fill level, queue depth against a hard consumer limit, replication lag against a stated RPO, and error-budget burn. These are wrong at any baseline, and an anomaly model would happily learn an unhealthy plateau as normal.
Move to baseline-driven detection where the metric is expected to vary with time of day or week and the absolute number is meaningless without context: CPU and memory on batch hosts, request rate, thread pool saturation, and latency on services with strong daily seasonality. Static CPU thresholds on hosts that legitimately run at 85-95% during the shipping window are the single largest source of the 638 notifications and should go first.
Two cautions. Anomaly detection needs a baseline long enough to learn weekly seasonality; for the two seasonal services, wait for at least four to six weeks of history or use a policy with an explicitly longer seasonal window, and keep a static ceiling as a backstop in the meantime. And anomaly detection on a per-host scope will still generate host-count-scaled noise, so scope policies to services or workloads wherever possible.
2. Design the three-week rollout and the metrics you would use to prove it worked.
Read the worked answer
Roll out in shadow mode first, then cut over by tier.
Week 1: inventory every condition and tag it static-invariant, seasonal-baseline or plain-noise. Delete duplicates. Fix scoping so policies target services and workloads rather than individual hosts, which removes host-count multiplication immediately at almost no risk.
Week 2: enable the candidate baseline policies in shadow mode so they evaluate and record but do not page. Compare their decisions against the static policies over the full shipping-window cycle, including a weekend.
Week 3: cut over the top-noise policies first (batch CPU, memory), keep the invariant thresholds, and leave one static backstop per critical service.
Metrics to report weekly: notifications per shift against the 638 baseline; share of notifications rated actionable by the responder; count of incidents detected by the new policies that the old ones missed; median time to acknowledge; and missed incidents, which is the guardrail metric that decides whether the cutover continues. Set a target such as fewer than 40 notifications per shift with no increase in missed detections. Report the twelve genuine incidents from the baseline shift as the regression test set.
3. The 40-minute checkout slow ramp is the hardest case. What detection strategy catches degradation that never breaches a threshold?
Read the worked answer
Detect the shape, not the level. A ramp that stays inside the static band is invisible to any level-based condition, so the signal has to be either slope, sustained deviation, or budget consumption.
- SLO burn-rate alerting is the strongest option. Define an objective on checkout p95 and alert on multi-window burn rate. A 40-minute degradation erodes budget faster than the baseline even when no absolute number is breached.
- A derivative or change-over-time condition on p95 compared with the same window in prior weeks catches the ramp directly, provided the baseline window covers the seasonal cycle.
- A lower-threshold condition with a long duration, for example p95 above a modest value for 30 minutes, is cruder but easy to explain and tune.
Whichever is chosen, validate it against replayed historical data from the incident so the team knows it would have fired, and record the expected detection latency. Also fix the attribution gap: if the ramp was visible only in aggregate but not tied to a service, entity scoping must be corrected before the new condition is trusted.