Halving APM and log ingest on a 180-service estate without losing incident explainability
An online retailer instruments 180 services with Elastic APM on Elastic Cloud Hosted. Ingest is 34 TB/month of traces and 61 TB/month of logs, Observability spend grew 40% quarter over quarter, and the platform team has been asked to cut ingest roughly in half without losing the ability to explain customer-facing incidents. Tracing is head-based at 100% for every service, including a catalogue browse endpoint that produces 62% of all spans and has a 0.4% error rate. Payments and checkout produce 3% of spans but own the 99.9% availability SLO. Separately, the on-call rota receives about 900 alerts per week, mostly from static latency-threshold rules on individual services, and two real incidents last month were reported by customers before any alert fired. Logs land in logs-* data streams under a single ILM policy: hot for 30 days, then delete. Compliance requires 13 months of retention for a subset of logs-audit-*.
1. Design the trace-sampling change that removes about half the trace volume while keeping the SLO-owning services fully observable. State explicitly what you lose and how you compensate.
Read the worked answer
Where the volume is. The browse endpoint is 62% of spans with a 0.4% error rate; payments and checkout are 3% of spans and carry the SLO. Head-based sampling is decided at the trace root and propagated, so controlling root services controls total volume - there is no need to sample uniformly and hope.
Change. Set transaction_sample_rate to 1.0 for payments, checkout and order-service; 0.02-0.05 for catalogue browse; 0.1 for the uncatalogued middle tier. Apply it through APM agent central config from Kibana/Fleet so 180 services change without a redeploy. Java agents can additionally use adaptive sampling, which targets a spans-per-minute budget per agent and is the practical way to cover services nobody has manually classified. Then cut noise before it becomes spans: transaction_ignore_urls for health checks, readiness probes and static asset handlers - these frequently account for a large share of span count and carry no diagnostic value - and cap per-trace size with transaction_max_spans so a fan-out bug cannot emit thousands of spans per trace.
What you lose. A sampled-out trace is not recoverable, and APM error views will only reflect sampled traces. You compensate by keeping 1.0 on the low-volume services where errors are rare but fatal, accepting 2-5% on browse where errors are frequent enough to be statistically visible, and keeping error-focused log rules as a second signal. Expect the reduction to be dominated by root-service rates rather than a uniform multiplier - model it from per-service span counts before promising 50%. Before designing around tail-based sampling, verify it is available in your specific deployment type and version.
2. Rebuild the alerting so the rota gets fewer, more meaningful pages and so a fast hard outage is caught before customers report it.
Read the worked answer
900 alerts per week is about 2.5 per hour on a rota that cannot act on them, and the two missed incidents show the rules watch the wrong signal.
Layer 1 - symptoms. Create SLOs on the checkout journey (99.9% over a 30-day rolling window) and alert on burn rate with multiple windows: page when the 1-hour burn rate exceeds 14.4x (roughly 2% of the 30-day budget consumed in one hour) and a shorter window confirms it is still burning; open a ticket when the 6-hour burn rate exceeds 6x. This fires on user-visible pain regardless of which dependency caused it, and it stays quiet during normal variance in a way no fixed latency threshold can.
Layer 2 - causes. Convert per-service latency and error rules into baseline-relative signals (latency above the service's own 28-day p95, error rate above baseline, saturation of the dependency the SLO depends on) and route them to a dashboard and daily digest rather than the phone.
Reduce the count mechanically. Group alerts by service and by SLO rather than per host; add alert delays and flapping detection so a five-minute spike does not page; delete threshold rules that duplicate an SLO.
Close the gap that let customers notice first. A 1-hour burn window will not catch a three-minute hard outage, so keep one fast rule with a 1-2 minute window on the checkout entry point's error rate. Target under 40 actionable pages per week and review every page that needed no action.
3. Design the retention and volume strategy for logs, given 61 TB/month, a 13-month audit requirement on a subset, and a possible future move to Serverless.
Read the worked answer
Split the requirement. One policy is currently serving two different obligations. Give logs-audit-* its own index template and ILM policy: hot 3 days (rollover at 50 GB or 1 day), warm at 3 days with forcemerge to one segment, best_compression and one replica, cold at 30 days, frozen at 90 days, delete at 400 days. Frozen uses partially mounted searchable snapshots, so the 13-month copy lives in object storage at a fraction of hot cost - this needs an Enterprise-level licence, so confirm entitlement before designing around it. Everything else: hot 7 days, warm 30, delete at 60. That second bucket is where most of the 61 TB actually goes.
Cut at the source, not only at the tier. The usual culprit is DEBUG logging from chatty services. Add an ingest pipeline with a drop processor keyed on log.level for named data streams, keeping DEBUG for the services currently being debugged, and collapse repeated multi-line stack traces with a fingerprinting step. Tiering moves cost; dropping moves cost and query load.
Serverless caveat. On Serverless, ILM is not a control surface you own - lifecycle and retention are largely platform-managed. That makes this split an argument for keeping audit data on Hosted, or for verifying exactly which retention controls the Serverless project exposes before committing the audit corpus to it.