420 Alerts a Week and Nobody Looked

Quantification. 420 alerts a week is ~60/day; 89% self-resolve, leaving ~46 actionable alerts a week against 420 pages of noise - roughly 9:1 noise to signal. Twelve night pages a week across five engineers means someone is woken every other night, which is the mechanism by which a real page gets ignored. Two of the three characteristics of a good alert are missing: it is not symptom-based (node CPU is a cause, not a user-visible effect) and it is not tied to a defined user journey. The synthetic probe from one region measures reachability, not checkout latency, so it structurally cannot see this incident.

Model. Define an SLI per user journey as good events ÷ valid events - for checkout, POST /checkout returning success and rendering within 800ms over all checkout requests. SLO 99.9% over 30 days gives 43.2 minutes of budget. Alert on multi-window burn rate:

• 14.4× over 1h and over 5m → page. 14.4 × 1h ÷ 720h = 2% of the monthly budget in one hour.
• 6× over 6h and over 30m → page (or page during business hours, ticket otherwise). 6 × 6 ÷ 720 = 5%.
• 1× over 3d and over 6h → ticket. 3 days at 1× is 10% of budget; not urgent, but it catches slow bleeds that never trip the fast windows.

The two-window requirement (long window for significance, short window for reset) is what stops a resolved incident from continuing to page.

Checkout-specific symptom alert. Page when p99 of POST /checkout exceeds 800ms for 5 minutes and success rate drops below baseline - a direct expression of user pain that fires in minutes independent of the SLO. Route CPU and node alerts to a dashboard or a low-severity ticket; they are diagnostic context, not pages.

Diagnosis. user_id and request_id are unbounded. Series count grows with distinct label values, so 4.2 million active series from one metric, each with head-block memory overhead, is the direct cause of the weekly OOM and of scrape intervals slipping - the scrape cannot complete before the next is due, which is also why two services silently stopped being scraped.

What to drop. Both labels, at the source. request_id has no analytical value in a metric at all - it belongs in logs and traces, where it is the join key. user_id is worse than useless in aggregate: it makes per-user queries possible but no aggregate query faster, and it is a privacy liability if the label value is an email or account number.

How.

1. Change the instrumentation in the application to stop emitting those labels; use exemplars to link a metric sample to a trace ID instead.
2. On the ingest side, add metric_relabel_configs with labeldrop: [request_id, user_id] so existing exporters and third-party metrics are cleaned immediately - this fixes the problem today, not next sprint.
3. Normalise route labels: /orders/8842/items must become /orders/{id}/items, otherwise the path label becomes a second unbounded dimension. This is often the hidden second cause.
4. Reduce histogram buckets to those that matter for the SLO boundaries (e.g. 100ms, 250ms, 500ms, 800ms, 2s) instead of fifteen buckets per route.
5. Add recording rules for the dashboard queries so 140 dashboards do not each scan raw series, and set a series budget per service with an alert on prometheus_tsdb_head_series growth.

What you lose. Per-user and per-request slicing directly in PromQL. You keep it through traces and logs, joined by trace ID, which is the right tool for that question. You also lose some long-tail histogram resolution; pick the boundaries around the SLO and accept coarser detail elsewhere. Net effect: the scrape completes, the OOM stops, and the two unscraped services come back into monitoring.

Three complementary methods, cheapest first.

1. Backtest against history. Point the proposed recording and alerting rules at the retained Prometheus blocks covering last month, or use promtool test rules with recorded fixtures. Then measure two numbers: did the new rules fire during the known 90-minute checkout degradation (recall), and how many times did they fire when nothing was actually wrong (precision). A rule set with perfect recall and 40 false pages has not solved the problem.

2. Shadow mode. Run the new rules for two weeks with PagerDuty routing disabled and output going to a dedicated channel. Compare against what actually paged. This catches rules that are technically correct but fire constantly during normal deploy churn - the most common failure mode of a freshly written burn-rate rule, because a deploy that briefly drops traffic can spike a ratio.

3. Game day. Inject latency into the checkout path in staging using a proxy or a feature flag, and confirm end to end that the symptom alert pages within five minutes, reaches the correct rotation, carries a runbook link, and that the on-call can identify the failing dependency from the dashboards. Measure time-to-detect and time-to-correct diagnosis, not just "did it fire". Run the exercise with the person who was on call during the real incident.

Track after rollout: alerts per week (target a steep reduction from 420), percentage of alerts acknowledged as actionable, pages per night (target ≤ 2), time to acknowledge, time to detect for real incidents, false-positive count per rule, and SLO compliance per service. Also track series count and scrape duration to confirm the cardinality fix held. Review the top five noisiest rules monthly and delete or demote anything that has not produced an actionable page - an unowned alert is the same as no alert.