A $180k observability bill and 2,300 mostly-useless alerts

Order by savings per unit of blast radius, and start with the data nobody queries during an incident.

1. Drop debug-level ingest (55% of log volume). 14 TB/day × 0.55 = 7.7 TB/day removed. If log cost tracks volume, that is up to 55% of $95k ≈ $52k. Risk: someone's incident playbook greps a debug line. Mitigation: move debug to a sampled or on-demand path first, confirm query logs show no reliance, then drop; keep error and warn untouched.

2. Tier log retention. 82% of queries hit 24 hours, and only 3% of volume is ever queried. 7 days hot plus 30 days on searchable snapshots or object storage typically removes 50–70% of the remaining cluster cost. After step 1 the log bill is ~$43k; tiering plausibly saves $20–25k. Risk: slow queries against cold tiers during incidents. Mitigation: publish the tier boundary and pre-warm the top runbooks.

3. Tail-sample traces instead of 100% head sampling. 900M spans/day at $40k. Keeping 100% of error and slow traces plus ~10% of the rest cuts volume by roughly 85%, about $34k. Risk: lost coverage for rare failures - mitigated because errors and slow traces are exactly what the tail policies retain.

4. Metrics last, and selectively. "60% of series never queried" is a trap: recording rules and alerting expressions query series that no human ever opens in Grafana. Before dropping anything, join the unused list against every recording rule, alert expression, and dashboard. Realistic saving $10–15k, with the highest risk of breaking detection.

Steps 1–3 alone reach roughly $105k, comfortably past the 40% target ($72k), so metrics work can be done carefully rather than urgently. Do not do all four at once - stage them across the quarter so an incident can be attributed to a specific change.

Protect by category, and write the categories into policy rather than trusting memory.

• SLO and error-budget data. The metrics that back each team's SLOs, plus the alerting series. Cutting these does not reduce observability cost, it removes the ability to know what availability you bought. Record the retention and resolution requirement as part of the SLO definition.
• Security, audit, and financial records. Payment and access logs usually carry a statutory retention period that has nothing to do with debugging. These should never have shared a cluster with debug output; separate them and keep them out of all future consolidation proposals.
• Error and slow traces. These are the highest-value, lowest-volume slice. Any sampling policy that drops them is a false economy - and note that they are also the cheapest slice to keep.
• Deploy and change markers. service.version on spans and deployment annotations on dashboards cost almost nothing and are the first thing you want at minute three of an incident.

Protection mechanisms that actually hold up:

1. Cost allocation per team, so the team generating 55% of log volume sees it on their budget. Central budgets hide this until finance intervenes, and then the cuts arrive indiscriminately.
2. An explicit allowlist with an owner for anything exempt from retention reduction. Exemptions with a named owner get renewed; exemptions labelled "important" do not.
3. A coverage check after every cut: for the top five incident runbooks, confirm each query still returns data at the expected resolution. Run it as a scheduled check, not a one-off, because retention policies drift.
4. Query-driven retention, recomputed quarterly from actual query logs. "Nobody queries this" has a shelf life of about one incident.

The core problem is that 94% of pages train people to ignore pages. With 2,300 alerts per quarter, roughly 25 per day, an on-call engineer who dismisses 94% of them will eventually dismiss the one that mattered - and a 22-minute median time-to-detect suggests the actionable ones are not surfacing promptly either.

What to change:

• Page on symptoms, not causes. Page when an SLO is being burned or a user-facing capability is failing. Cause-based signals - CPU high, queue depth rising, a pod restarting, disk 70% - become tickets or dashboard annotations. Any alert whose only content is "a resource metric crossed a threshold" is not a page.
• Use multi-window burn-rate alerts. A fast-burn condition (for example a 1h window burning budget at 14× against a 5-minute short window) catches acute outages without flapping; a slow-burn condition (6h at 6×) catches gradual regressions. Both are expressed against the error budget, so the alert and the SLO cannot drift apart the way independent thresholds do.
• Delete or downgrade anything that has been non-actionable across two review cycles. Route surviving low-urgency signals to a ticket queue with an owner and a due date, not to a human at night.
• Fix the 22 minutes separately. Detection latency is usually a symptom of missing SLIs at the right boundary or of alert routing, not of too few alerts. Adding alerts will not help; adding one well-placed SLI will.

Metrics that prove it worked - pick them before you start:

• Actionable rate per alert (target above 50%, reviewed monthly on a sample).
• Pages per on-call shift and pages outside working hours.
• Median time-to-detect for the top customer-facing journeys, measured from the user-visible symptom, not from the first infrastructure signal.
• Percentage of incidents first detected by an alert versus reported by a customer - the number that actually says whether observability is working.

Expect the alert count to fall by an order of magnitude and time-to-detect to improve; if only the count falls, you removed signal rather than noise.