Alarm Fatigue Hiding the One Signal That Matters
An operations team has 1,400 CloudWatch alarms. Most are static-threshold CPU and memory alarms tuned years ago. The on-call rotation receives ~90 pages per week; roughly 85% are dismissed as noise within minutes. Mean time to acknowledge a real incident has crept up to 22 minutes. Last month a genuine outage - elevated 5xx on the checkout service - was paged but sat behind a burst of CPU alarms on unrelated batch hosts and was acknowledged 40 minutes late. Leadership wants "fewer pages" but is worried about missing real incidents. The team also has no SLO defined for checkout, and logs are retained 7 days with no structured fields.
1. Diagnose why the real incident was missed and propose an alarm strategy that reduces pages without increasing risk.
Read the worked answer
The real incident wasn't missed for lack of alarms; it was buried by them. 90 pages/week at 85% noise trains responders to triage fast and shallow, and a 5xx page competing with a CPU-alarm burst loses. Static CPU/memory thresholds are poor proxies for user impact and generate noise without predicting outages.
Strategy:
- Page on symptoms, not causes. Define an SLO for checkout (e.g., 99.9% success) and page on error-budget burn rate (fast-burn multi-window alerts) rather than raw CPU. CPU/memory become dashboards or low-severity tickets, not pages.
- Tier alarms by actionability. Page only what a human must act on now; route the rest to tickets/chat. Delete or downgrade alarms nobody has acted on in 90 days.
- Deduplicate and group related alarms so one incident is one page, and ensure the checkout symptom alarm has clear priority/routing.
Fewer, higher-signal pages directly reduces the 22-minute ack time because responders aren't context-switching through noise.
2. The team lacks SLOs and structured logs. What must be built first to make symptom-based paging possible, and in what order?
Read the worked answer
You can't burn-rate alert without an SLI, and you can't compute a reliable SLI without usable telemetry.
Order:
- Define the SLI and SLO for checkout - e.g., proportion of successful checkout requests (2xx/3xx over valid requests) measured at the edge/ALB, with a target and window. This is a decision, not tooling; do it first.
- Make the telemetry trustworthy. Emit structured logs with request IDs and outcome fields, and/or rely on ALB/access metrics plus application metrics for the success ratio. Extend retention beyond 7 days for investigation. Without structured outcome data the SLI is a guess.
- Instrument the SLI as a metric and build multi-window burn-rate alarms (e.g., fast 1h/5m and slow 6h/30m) so pages fire on real budget consumption.
- Then retire the static CPU/memory pages, converting them to tickets/dashboards, and verify over a few weeks that no real incident went unpaged before deleting them entirely.
Retention and structured logging aren't optional cleanup - they're prerequisites for both alerting and the postmortems that justify the new thresholds.