Twelve Million Series and a One-Percent Sample

• Head sampling at 1% samples the wrong traces. The keep/drop decision is made before the outcome is known, so you retain 1% of boring fast successes and 1% of failures. Across 400 services, the rare failing path that matters is exactly what gets thrown away.
• Unbounded cardinality is eating the budget. Per-pod x per-URL-path x histogram is the classic bomb: raw paths containing IDs produce series nobody alerts on. Six of every ten series come from thirty services, and 80% ingest utilization means the next service to launch is an outage risk.
• Logs are the primary diagnostic tool. Grepping for a request ID supplied by support means the data exists but is unnavigable. Unstructured logs cannot be joined to traces or metrics without a trace_id.
• No SLO-based first signal. With no burn-rate dashboards per user journey, detection starts with a customer or a support agent. A 70-minute time-to-identify with 140 deploys a week is unsurprising: nobody can ask "what changed" against a symptom.

In short, the money buys storage volume, not answerable questions.

1. Tail-based sampling with an explicit policy - keep 100% of traces with errors, 100% above the p99 latency threshold, and a sample of the rest. Target 10–15% of current volume. Effect: failing paths become traceable for the first time while trace volume falls, so cost and usefulness move in the right directions simultaneously.
2. A cardinality budget per service - route templates instead of raw paths, drop pod-level dimensions from anything that is not a per-pod alert, cap series per service and fail the observability CI check when exceeded. Effect: roughly 12M → 4M series, ingest headroom restored, and the remaining series become legible enough to alert on.
3. Standardize telemetry in the service template and test it - structured JSON logs carrying trace_id and span_id, RED metrics and a generated SLO burn dashboard, with exemplars on the latency histogram so a spike links straight to a sampled trace. Observability-driven development: the service's test suite asserts the expected span and log fields, so a new service cannot be born unobservable.

Sequencing matters. Do the cardinality work first because it is the hard constraint; then tail sampling, which depends on having headroom; then the template, which only sticks once the first two make the right thing the cheap thing.

• Replay the last three incidents. Take the known culprit service and ask how long it takes to find with the new setup. That is a concrete before/after instead of an opinion.
• Measure detection separately from identification. Target: the SLO burn alert fires before a customer report in at least 80% of SEVs, and the culprit span is identified in under 5 minutes.
• Template adoption: percentage of newly created services emitting compliant telemetry on day one. If this stays low, the standard will rot.
• Cost per useful trace, not raw cost - total spend divided by traces actually opened during an incident. Spend down 30% while traces viewed doubles is an unambiguous win.
• A recurring game day. Inject a 5% error rate into a dependency and time detection through identification, quarterly. If identification time regresses, the instrumentation is decaying and you catch it before the next SEV-1 does.