An SLO nobody was alerted on, and debugging logs nobody can afford

Page on what the customer feels, not on what the machine is doing. A CPU spike that no user noticed is a cause-based alert; it trains on-call to ignore the pager, which is exactly how a four-hour outage goes unremarked.

Build:

1. Define a Cloud Monitoring SLO of 99.9% over a rolling 28 days with a request-based SLI: good requests = non-5xx, measured from load balancer logs or the service's own request metric. A latency SLI alongside it if latency is contractual.
2. Alert on burn rate, not a raw threshold. Multi-window: a fast-burn policy at ~14.4x over 1 hour pages immediately (a total outage surfaces in minutes), and a slow-burn policy at ~6x over 6 hours opens a ticket. A single error rate > 1% for 5 min rule both pages on blips and misses a slow decay.
3. Add an uptime check from three or more regions against an endpoint that exercises the database, not a static health page.
4. Route to a notification channel with a documented escalation, and test it by breaking something deliberately.

Diagnosis of the miss: the outage was found by a customer, so either no alert existed or it watched an endpoint that stayed healthy while real requests failed. An SLI from real traffic is the only thing that cannot be fooled that way.

Two facts drive the design: exclusion filters do not reduce what log-based metrics see (metrics are evaluated on the stream before exclusions), and exclusions are not retroactive.

1. Fix the source first. Ship a log-level env var and turn the debug statement off in production. The exclusion is a safety net, not the fix; if the statement stays, someone re-enables debug during an incident and the bill returns.
2. Add an exclusion filter on the _Default sink, e.g. severity=DEBUG AND resource.labels.namespace_name="prod". Because metrics are computed first, any counter you already rely on keeps working.
3. Keep a lane for debugging. Rather than sampling inside the exclusion (you cannot), create a user-defined sink routing DEBUG from production to a Cloud Storage bucket with a 30-day lifecycle. Cheap, queryable in BigQuery if you need it, and not in the hot path.
4. Tune retention, not ingestion. Set _Default to 30 days. Do not touch _Required.

Measure before and after by log volume per log name in Logging metrics; a blanket exclusion on a shared sink is how teams silently delete the one log they needed.

There are two separate gaps: history and prevention.

History: Kubernetes events in a cluster are kept for about an hour by default, which is precisely why the team kept missing them. GKE forwards events to Cloud Logging when system logging is enabled, under the k8s_cluster resource. Enable it, create a log-based metric on reason="Evicted", and alert on it. This costs almost nothing and closes the gap permanently. Container memory is separately available as kubernetes.io/container/memory/used_bytes with memory_type=working_set; watch node allocatable usage for the pressure signal itself.

Prevention: pods without memory requests land in BestEffort QoS and are evicted first; pods with requests below limits are Burstable and still go before Guaranteed ones. Set requests equal to limits for the tracking service, size them from a week of VPA recommendations in recommend-only mode, and let the cluster autoscaler (or node auto-provisioning) add capacity for the Guaranteed pods rather than letting the node trim them.

Alert on eviction count rising, not on a single eviction: some churn is normal and a noisy pager gets muted.