3.8 TiB a day and nobody can find the slow request

Arithmetic first: 3.8 TiB/day is about 3,891 GiB/day. At roughly $0.50/GiB ingested, that is about $1,946/day, near $58k/month. Nothing else in the observability budget matters until this is addressed.

The mistake to avoid is deleting logs by severity. Reduce by asking what a query ever needs. In a service at 9,000 rps, per-request debug lines are 85–90% of the volume and are useful only for one request at a time - which is what traces and error-context logs are for.

Design:

• Exclusion filters at the log router for debug/info lines on the hot ingest path, keeping warnings and errors.
• Keep full request logging at a sampled rate (e.g. 1 in 200) with the request ID, so a customer report can still be correlated.
• Log-based metrics for the rates you actually alert on (requests by status by region), so counters survive the exclusion.
• Split buckets by retention: 30 days for application logs, 400 days for audit, 7 days for high-volume debug.

Expected result: 3.8 TiB/day down to roughly 0.4 TiB/day, near $194/day, with error triage unaffected. Validate by replaying last month's top five incident queries against the new configuration before you delete anything.

Cardinality is a design decision, not a quota problem. 240,000 distinct label values is 240,000 time series for one metric, and every query must scan them; that is why dashboards time out.

Move the identifier out of the metric. Vehicle-level questions ('what did van 4471 do between 14:00 and 14:05') are a trace or log query, not a metric query. Keep a bounded metric: aggregate to region, fleet, software version, hardware revision - labels whose value count is in the tens, not the hundreds of thousands. Then use exemplars to link an aggregated latency bucket back to a specific trace when one is slow.

If per-vehicle state genuinely must be charted (e.g. a top-N noisy-vehicle report), compute it in the pipeline and export a ranked top-100 gauge plus a total count, rather than exporting every series and sorting at query time.

General rules to write into the service template: no unbounded identifiers (user ID, request ID, vehicle ID, URL path) as metric labels; cap label cardinality in CI by asserting on the metric descriptor; set the metric's series limit deliberately. Enforce it in code review, because Monitoring will happily accept the write and only fail at read time.

Normal CPU and memory with 8x p99 and a normal error rate says: requests are completing, slowly, and it is not local resource saturation. That rules in the dependency path, the network, or scheduler throttling.

Order of investigation:

1. Split the latency by dimension. Is it all requests or one zone, one version, one tenant, one van firmware? Open the latency distribution, not just p99 - if p50 is flat and only p99 moved, you have a tail problem (queueing, one bad backend), not a capacity problem.
2. Use traces. At 9,000 rps, 1% head sampling is still ~90 spans per second - plenty. Compare the client span duration to the server span duration for the same request: a large gap is network, DNS, or queueing before your handler; a large gap between your span and the downstream call span names the dependency.
3. Check CPU throttling, not just CPU utilization. In Autopilot, a container hitting its CPU limit shows low utilization and high container_cpu_cfs_throttled_seconds. This is the classic 'all dashboards green, everything slow' signature.
4. Check the deploy timeline and the dependency's own SLO for the three-hour window.

Then fix the detection, not just the incident: no alert existed on p99 against the SLO's latency bound. Add a burn-rate alert on the latency SLI, and switch to tail-based sampling so 100% of slow requests are retained while normal traffic stays at 1%. Head sampling is the reason a 3-hour tail event was invisible in traces.