Head-Based Sampling Is Eating the Errors
Paymob Payments runs 220 services on Java, Go and Python, peaking at 18,000 requests per second. A previous engineer set DD_TRACE_SAMPLE_RATE to 0.10 globally at the tracer to control cost. Over the last six weeks on-call investigated two production incidents - a 5xx spike on /v1/charge and a p99 regression on /v1/refund - and in both cases the trace explorer showed almost nothing for the failing window. Leadership now wants complete traces for errors on the ten tier-1 services, per-endpoint p99 latency for capacity planning, and 15-day investigation of traces tagged customer_tier:enterprise, all within a flat budget.
1. Traces were never deleted from storage, so why is the explorer empty for the failing window? Explain the mechanism and what has to change.
Read the worked answer
The sampling decision is made before the error exists. DD_TRACE_SAMPLE_RATE: 0.10 at the tracer is head-based: the rate is applied when the trace starts, before the code knows it will return a 500. 90% of traces are discarded at the source. Nothing downstream - retention filters, the trace explorer, a search for status:error - can recover a span that was never sent. The failing traffic is in the missing 90%, which is why the explorer looks empty for exactly the window that hurt.
Two levers that are routinely confused:
- Ingestion controls (tracer/Agent head-based sampling) decide what Datadog ever receives.
- Retention filters decide which ingested spans stay queryable for the retention period (default 15 days), as opposed to live search, which covers only the last 15 minutes.
Retention filters act only on ingested data. So the fix is to raise ingestion for the services that matter and enable error and rare-trace prioritisation in the tracer/Agent so those traces survive regardless of the base rate - then use retention filters to index them.
2. Design the ingestion and retention configuration across the 220 services so tier-1 errors are complete, enterprise traces are searchable at day 10, and cost stays flat.
Read the worked answer
Budget ingestion by service, not globally.
- Use the APM ingestion view to get requests/second and span volume per service. Take the ten tier-1 services to 100% ingestion - estimate their share of the 18k rps first; if they are 30% of traffic, this alone may fit the budget.
- Leave chatty, low-value services at 10-25%, with error and rare traces prioritised so you don't repeat the outage-blindness failure there either.
- Configure retention filters so the default index keeps error spans and spans where
customer_tier:enterprise. That delivers the 15-day investigation the business asked for. Everything else falls out of the index. - Remember live search: the last 15 minutes at full fidelity is what most on-call debugging actually uses and carries no retention cost.
Verify by replaying a known failure - find a recent 500 on /v1/charge, confirm the complete trace is present with all spans, and confirm an enterprise-tagged trace is retrievable at day 10 rather than assuming the filter matched. Check the enterprise filter's tag exists on the entry span, or it will match far less than expected.
3. How do you provide per-endpoint p99 for capacity planning without paying to index every span?
Read the worked answer
Capacity planning needs aggregates, not individual traces. Create a span-based custom metric: a distribution of span duration tagged by service, resource_name (the endpoint), and env. Dashboards and capacity alerts read its p99/p95 directly.
Why this is the right tool: indexed spans are the most expensive place to keep high-volume data, and you only need the tail percentiles from them. A distribution metric gives you percentiles at metric-cardinality cost, keeps 15 months of trend for sizing, and lets you compare this month's /v1/refund p99 to last quarter's.
Caveats to state explicitly:
- The metric is computed from ingested spans, so the endpoints you size capacity for must be at 100% ingestion - sampling at 10% would give you a percentage of a sample, which is fine for trends but dangerous for SLO-adjacent sizing.
- Keep the tag set tight (
endpoint,service,env, status class). Adding a user ID turns a useful metric into another 3M-series incident. - Indexed spans remain the tool for why a specific slow request was slow - use them for investigation, not for sizing.