OTCA: Tail Sampling Drops the Errors You Care About

Tail Sampling Drops the Errors You Care About

A payments team runs the OTel Collector as a gateway Deployment (3 replicas) behind a load balancer. All 40 services send OTLP/gRPC directly to it. They added a tail_sampling processor to keep 100% of traces with status.code = ERROR and 1% of everything else, followed by batch. Error budget dashboards show 0 error traces even though the p99 error rate on the upstream HTTP metrics is 2.3%. The processor config reads: policies: [{name: errors, type: status_code, status_code: {status_codes: [ERROR]}}, {name: sample, type: probabilistic, probabilistic: {sampling_percentage: 1}}]. Traffic is ~12k spans/sec across 40 services, and the collector pods sit at ~85% CPU with frequent OOMKills at their 1Gi limit.

1. The error traces are being generated upstream but never reach the backend. Identify the two structural reasons in this deployment that break tail sampling, and explain why each causes the loss.

Read the worked answer

Reason 1 - Multiple gateway replicas without a load-balancing exporter. Tail sampling requires that all spans of a given trace arrive at the same collector instance so the decision is made once, after the whole trace is seen. With 3 replicas behind a plain LB, spans of one trace are scattered across pods. Each pod sees a partial trace, its decide step fires on a timeout or on partial data, and error spans that land on a pod that never sees the root may be sampled out at 1%. The fix is to put an OTel loadbalancing exporter in front (routing by traceID), so a trace's spans are consistently hashed to one tail-sampling collector.

Reason 2 - Resource exhaustion corrupting decisions. At 85% CPU and OOMKills, the in-memory tail_sampling decision cache and pending-trace buffers are evicted. A restart drops all in-flight traces; those traces are never exported at all, and error traces are disproportionately lost because they are the ones the sampler is holding longest waiting for the full trace. Memory pressure is worst precisely on high-cardinality error traces.

Diagnosis path: check otelcol_processor_tail_sampling_count_traces_sampled by sampled/policy and otelcol_processor_tail_sampling_sampling_trace_dropped_too_early - a high too-early drop rate confirms the scatter problem. Also confirm with otelcol_processor_refused_spans during OOM windows.

2. The team proposes simply raising the memory limit to 4Gi and adding a 4th replica. Evaluate whether this fixes the problem, and describe the correct architecture.

Read the worked answer

It does not fix it. More memory delays the OOM but the scatter problem is fundamental: adding a 4th replica makes it worse, because the probability that all spans of a trace reach one pod drops further. Tail sampling is a stateful, trace-affine operation; horizontal scaling of the sampling tier only works when trace affinity is enforced upstream.

Correct architecture:

Keep a stateless collector tier (gateway) that does batching/redaction and fans out to a load-balancing exporter configured with routing_key: traceID (or service for a coarse split), pointing at the tail-sampling tier.
Run the tail-sampling tier as a separate Deployment; scale it, but route by traceID so each trace is decided by exactly one pod.
Size memory for the sampling num_traces window: set tail_sampling.num_traces explicitly (e.g. 50k) and decision_wait (e.g. 10s) to bound memory, rather than relying on the default.
Optionally add a memory_limiter processor as the first processor in every pipeline so the collector sheds load gracefully instead of OOMing.

Trade-off to state: routing by traceID means a hot trace can still land on one pod; the load-balancing exporter uses consistent hashing so distribution is even at scale, but you must monitor per-pod span rates.