Alert Noise and the Missed Memory Leak
Platform engineering owns alerting for 125 services. There are 340 alert policies generating about 1,100 notifications a week, and 71% are closed within ten minutes with no action taken. The payments API has a static condition on CPU above 80% that fires every night between 01:00 and 03:00 during a batch settlement job, and the team has muted that policy indefinitely. Last month the same service leaked memory: error rate climbed from 0.2% to 4.1% over six hours while p95 latency stayed at 620 ms, below its 800 ms static threshold, and CPU remained above 80%, so the only policy that could have fired was the muted one. The service also has a baseline condition on throughput that pages on every marketing campaign launch, because campaign traffic looks anomalous against the trailing baseline.
1. Diagnose why this policy set both over-alerts and misses incidents, and propose a condition design for the payments API.
Read the worked answer
Two different failure modes live in one policy set.
Over-alerting: a static CPU above 80% condition on a service with a known nightly batch job encodes an expectation that is false for two hours every day. The mute is a rational response to a badly designed condition, not a discipline problem.
Missed incident: error rate was the signal that moved, and nothing watched it. Latency was the wrong target for a memory leak, and CPU was already pegged inside a threshold the team had stopped trusting.
Design for the payments API:
- Page on error-rate saturation: an NRQL condition on error percentage, evaluated over a window short enough to catch a six-hour ramp early, with a warning tier at roughly 1% and critical around 5%.
- Page on memory pressure rather than CPU: heap used after garbage collection, or container memory working set against limit, so a leak is a leading indicator instead of a postmortem fact.
- Keep one static guardrail that must never be crossed regardless of learned baseline, such as availability below 99% or error rate above 5%, so a slow step change does not get normalized into the baseline.
- Use baseline conditions where traffic genuinely has recurring shape, and static thresholds where the limit is physical or contractual.
- Route to one incident per service, not one per policy, and require every policy to name an action and an owner. Anything without both should be deleted, not muted.
2. How should the team handle the known nightly batch spike without muting, and what governance should apply to mutes in general?
Read the worked answer
An indefinite mute is a promise to not be told about the exact thing you were worried about. Here it directly contributed to a six-hour undetected incident, so the fix is both technical and procedural.
Technical alternatives to muting:
- Scope the condition instead of the service. A baseline condition on CPU learns the 01:00-03:00 shape, so the nightly peak becomes normal while a deviation from that peak still pages.
- Exclude the expected signature in the query, for example by filtering out the batch job's metrics or namespace, while keeping the condition live for everything else on the host.
- If a time-window exclusion is genuinely necessary, make it self-expiring and visible in an audit list rather than a permanent mute.
- Add a no-data or gap-detection condition so a mute can never hide a total loss of signal.
Governance:
- Every mute gets an owner, a ticket, and an end date; anything muted longer than 30 days is treated as a design flaw and scheduled for rework.
- Track alert quality per policy, meaning the ratio of notifications to actual incidents. Policies with a long no-action history get fixed or deleted.
- Review the mute list in the same meeting where on-call pain is discussed, so removing a mute is paired with adding the signal that replaces it: here, error rate and heap after GC.
3. Baseline conditions fire on every campaign launch. How do you tune them without silencing real anomalies, and what are their limits around deployments?
Read the worked answer
Baseline conditions compare current values against a learned model of normal, including time-of-day and day-of-week shape. They are strong for metrics with stable seasonality and weak for deliberate step changes, because a campaign launch or a new deployment genuinely is not normal and looks statistically identical to an incident.
Tuning approach:
- Separate signal types. Keep baseline conditions on saturation and resource metrics with stable seasonality. For traffic, move the condition to a derived outcome such as error rate, errors per request, or queue depth per consumer. Those stay flat during a campaign if the system is healthy and spike only when it is not.
- Evaluate per instance or per consumer group rather than fleet totals, so extra traffic cannot mask a degraded subset.
- Pre-declare scheduled events. Use a longer baseline window that includes prior campaigns, or an explicit exclusion with an end date, rather than a mute.
- Treat deployments as known step changes. During the first hours after a release, lean on static guardrails and compare post-deploy to pre-deploy, so a degraded version is not quietly adopted as the new baseline.
Validate the change both ways: replay the last campaign window and confirm the policy would not have fired, then replay the last real incident and confirm it would have. A tuning exercise that only passes the first test has simply recreated the old blind spot.