Fewer Pages, Faster Recovery, Lower Observability Spend

Diagnosis. The two customer complaints that fired no alert are the decisive data point: the monitoring is measuring the system's internals, not the service it provides. CPU and memory are causes; playback failure is the symptom customers experience. Alerting on causes produces both the 85% noise (a CPU spike that no user noticed) and the blind spots (a dependency failing while every local resource looks healthy).

Instrument user journeys first. Define SLIs for the three or four journeys that matter: playback start success rate and time-to-first-frame, stream rebuffering ratio, payment/checkout success, and login success. Measure them at the edge and, where possible, synthetically from multiple regions, so a regional CDN problem is visible before customers call.

Set SLOs and alert on error-budget burn. For each SLI, agree a target over a rolling window (e.g. 99.9% of playback starts succeed over 28 days). Page on multi-window, multi-burn-rate conditions: a fast burn (a large fraction of the budget consumed in an hour) pages immediately; a slow burn (2% of budget in six hours) creates a ticket. This single change typically removes most of the noise, because a burn rate is by construction tied to customer impact, and it gives the team a shared, quantitative language for "how bad is this".

Stop paging on: per-pod CPU or memory thresholds (route to a capacity review), single-node disk warnings, non-actionable dependency timeouts that self-recover, and any rule that has not resulted in a human action in 90 days. Keep them as dashboards or tickets, not pages.

Keep cause-based signals for diagnosis - they are excellent for answering "why" once a symptom has paged, and useless as the trigger.

Method: build an alert inventory before deleting anything. Export all 640 rules with owner, last-triggered date, and - the column that matters - whether the last page produced an action (acknowledged and fixed, or improved diagnosis). Rules that have not paged in 90 days, or have paged without action, are candidates for deletion, aggregation, or demotion to ticket/dashboard. Expect this alone to remove the majority of the 200 weekly pages. Add routing tiers so that only symptoms route to the pager; everything else goes to a team queue in business hours.

Logs: 4 TB/day with 70% debug from three services. Attack in order of cheapness:

1. Change log levels in those three services at the source - debug lines should not be emitted in production by default. This is free and immediate.
2. Structured logging with sampling for high-volume success paths; keep 100% of errors and warnings.
3. Drop and redact at the collection pipeline, not in the application, so the rule is centrally maintained and auditable.
4. Tail-based sampling for traces: keep all traces that errored or were slow, sample the rest.
5. Retention tiers - hot for 7–14 days for debugging, cold/archive for the compliance period.

Where cutting becomes dangerous. Three places:

• Cutting the telemetry behind an SLO. If retention is too short, slow-burn detection and trend analysis stop working, and post-incident review loses its evidence. Protect the signals that feed the error budget and the audit trail; cut elsewhere.
• Cutting audit and security logs to hit a cost target, which creates a compliance and forensic exposure that dwarfs the saving.
• Cutting in a way that increases MTTD. The test is not "did the bill fall" but "did pages fall and detection stay the same or improve". Track MTTD and the percentage of customer-impacting incidents detected before a customer report, alongside spend.

Report savings as a by-product of a signal-quality programme, not as the goal - otherwise the easiest win is to switch things off and hope.

Reframe the problem for the VP: burnout and slow recovery are the same problem. A team averaging two night pages a week, with 85% of pages non-actionable, is being trained to wake up, triage noise and go back to sleep. That habit is exactly what lengthens detection when a real incident occurs - the 25-minute MTTD is not a tooling limit, it is a team conditioned to distrust the pager. And with two engineers already asking to leave, the retention risk is a reliability risk: tacit knowledge of a 120-service estate walks out of the door.

Changes:

1. Fix the pager first. Symptom-based, burn-rate paging (previous task) removes most night pages. Set an explicit target: no more than one night page per engineer per week, and every page must be actionable - if it is not, the rule is deleted at the next review.
2. Rewrite the review process. Hold reviews for every customer-impacting incident, not just severity-1, within five working days. Replace "which engineer caused this" with contributing factors, detection gaps, and why the guardrail that should have prevented it did not exist. Publish the reviews internally, including leadership's own decisions as contributing factors where true. The first review under the new template is the signal the organisation will actually read - one that names a person destroys the change permanently.
3. Make follow-through visible. Every review produces a small number of action items with owners and dates; review their closure rate monthly. Reviews that produce no change teach the team that reporting is pointless.
4. Spread the load. Add a secondary on-call, compensate night pages explicitly, and give engineers a toil budget (e.g. no more than 30% of time on repetitive operational work) so alert cleanup is funded rather than borrowed from feature work.

How to show it is working: pages per engineer per week; percentage of pages judged actionable; MTTD and the share of incidents detected before a customer report; review action-item closure rate; and the pulse-survey scores for on-call sustainability and "I would report a mistake". Present them as a trend over two quarters, and be explicit that a fall in page volume accompanied by a rise in customer-reported detection is a failure, not a success.