9 TB per Day: Log Tiering After a Sampling Miss
A SaaS provider ingests 9 TB/day of logs into an observability platform; 70% comes from a Kubernetes cluster running a chatty debug logger. The observability budget is fixed, and ingest cost has risen 30% month over month. Security needs 400 days of retention for authentication logs; engineering wants 7 days for application debug logs. A recent incident took 4 hours to diagnose because the relevant payment-service logs were sampled out by a 10% head-based sampling rule. The platform team proposes switching all logs to 100% ingest with tail-based sampling, but that would triple cost. Alternatively, they can use log processing rules to drop debug logs at the agent and keep metrics/traces. Authentication logs are low volume but must be immutable; payment-service logs are high volume and contain request IDs that also appear in traces.
1. Design an ingest and retention strategy that satisfies security and incident response while controlling cost.
Read the worked answer
Use tiering and source-level filtering, not uniform 100% ingest.
- Drop or heavily filter the chatty debug logger at the agent. If 70% of 9 TB is debug, removing most of it addresses the cost driver directly. Keep a small sampled debug stream for troubleshooting.
- Keep authentication logs at full fidelity and route them to immutable, low-cost archive storage for 400 days. They are low volume and have a compliance requirement.
- Keep payment-service structured logs at full fidelity for a shorter active window, such as 30 to 90 days, because they carry request IDs that join to traces. Then archive or aggregate them.
- Keep metrics and traces in the active observability tier and use trace IDs to retrieve the relevant logs on demand.
- Apply tail-based sampling to high-volume application logs so errors, slow requests, and unusual status codes are retained even when normal traffic is sampled.
Estimate the reduction before implementation: removing 70% debug volume brings 9 TB/day toward roughly 2.7 TB/day before indexing overhead. Security retention should be excluded from any aggressive sampling policy.
2. Why did the 10% head-based sampling rule fail the incident, and when is tail-based sampling appropriate?
Read the worked answer
Head-based sampling decides at the start of a request, before the system knows whether the request will fail, be slow, or be interesting. A 10% rule therefore drops roughly 90% of all requests, including the rare failing payment requests whose logs were needed. The team could not reconstruct the failure because the decision was made without outcome knowledge.
Tail-based sampling buffers spans and logs until the trace or request completes, then decides using outcome criteria: keep errors, high latency, specific status codes, or requests from selected tenants. That preserves the incident-relevant records while reducing normal-volume storage. It is appropriate for high-volume application logs where failures are rare and trace context exists. It is not a substitute for compliance retention, and it has operational costs: collector memory, buffering, and consistent trace propagation across services. Use it for application diagnostics, not for immutable authentication records.
3. What governance metrics and controls would prevent the next cost spike and observability gap?
Read the worked answer
Track per-service log volume, bytes per request, debug-to-info ratio, ingest cost allocation, sampling coverage, and retention compliance. Add an observability data-quality metric such as the percentage of incidents where the needed logs were present. Review these monthly with service owners.
Controls:
- Enforce log-level policy in CI and reject deployments that enable debug logging in production by default.
- Set per-service ingestion quotas and budget alerts, with an exception path for security logs.
- Require a logging review for new high-volume services.
- Test sampling and retention rules in a canary namespace before broad rollout.
- Audit authentication-log retention and immutability separately from application-log optimization.
Measure incident MTTR and missing-log rate after changes. If tail-based sampling is introduced, verify that error traces and slow traces are actually retained. The goal is not merely lower ingest; it is predictable cost with evidence that critical diagnostic and security data survives.