Tail sampling with round-robin routing is producing broken traces

Cause: tail sampling is a stateful, per-trace decision, and round-robin routing violates that. The gateway pod that receives the root span knows nothing about children that landed on the other five pods. Each pod decides independently on a partial trace: one pod may keep the root and drop the children, another may keep a child and drop the root. Result: fragmented traces, and error policies that fire on whichever fragment happens to contain the error span.

Fix: route by trace ID, not by round robin.

1. Move the loadbalancing exporter to the agent tier and set routing_key: traceID (the default). Each agent hashes the trace ID and sends every span of that trace to one gateway pod.
2. Ensure every agent resolves the same gateway endpoint set. Use a headless Service or explicit DNS so all agents compute the same hash ring. If half the fleet sees three pods and half sees six, spans scatter and you are back where you started.
3. Keep the gateway tier behind a plain service only for health checks; do not put a round-robin exporter or a TCP load balancer in front of it.

Caveats worth stating: changing the gateway pod count remaps the hash ring, so in-flight traces split during scaling. Scale in steps during low traffic and expect a brief anomaly window. Also confirm the agents are not also exporting directly to the backend, which would double-count and produce duplicate trace IDs with different span sets.

Finally, verify the split is not partly propagation: if a service fails to extract context, it produces a new root, which looks identical in the UI. Check for unusually many root spans per trace ID before and after the routing change.

The 45-second endpoint is a direct hit on decision_wait: 30s. Tail sampling holds a trace for decision_wait and then decides. If the failing span - the timeout, the error status, the log about the downstream failure - is created at second 45, the decision was already made at second 30 without it. Traces that time out are exactly the traces you most want, and they are systematically the ones that exceed the decision window.

What to check:

• Compare decision_wait to the p99 and max span duration of the slowest endpoints. If max duration exceeds the wait, the policy is blind above that duration by construction.
• Confirm what "error" means in the policy versus in the metrics. A policy matching status.code == ERROR will not match spans that only carry http.status_code = 500 or a 4xx the metric counts as an error. Align the policy's OTTL/attribute matching with the metric definition.
• Check for late-arriving spans being dropped without a decision, and whether the trace map is evicting early under memory pressure - an OOM-adjacent gateway evicts traces and returns "no decision", which silently lowers the sampling rate.

What to change:

1. Raise decision_wait above the slowest endpoint's max duration (60s here) - and accept the memory cost, which is the real constraint.
2. Add a policy so slow-but-successful traces are kept by duration, not just by error, since 45s requests are the ones under investigation.
3. Reduce the probabilistic rate if the resulting volume is unacceptable; 3% of many small traces is cheap next to 100% of the slow ones.
4. Monitor effective sampling rate per policy, not just overall, so a policy that stops matching is visible as a metric rather than as a gap during an incident.

Rough estimate. At peak, 2.4M spans/min = 40,000 spans/s. With a 30s decision window the gateways hold roughly 40,000 × 30 = 1.2M spans in memory at steady state, spread across six pods ≈ 200k spans per pod. At about 1–1.5KB per decoded span with attributes, that is ~200–300MB of live span data per pod, and a Go process needs roughly 2× that as headroom before GC pressure stabilises. So on a 1Gi limit you would be at 50–60%, not 85–90%.

So the gap needs explanation, and the likely causes are:

• Uneven hashing. A few very large traces (fan-out of hundreds of spans) or a hot tenant can concentrate spans on one pod. Look at per-pod span ingestion rate, not the aggregate.
• Traffic peaks above 2.4M/min. Size for peak, not average; a 2× peak doubles the buffer.
• num_traces pre-allocation. The processor pre-allocates the trace map (default 50,000 traces per instance); large values across six pods are real memory before any spans arrive.
• Late spans and multiple copies. Spans arriving after a decision are held or re-queued, and the batch processor's queue adds another buffer.

Levers, in order: raise the memory limit only after fixing uneven distribution; increase pod count to spread the hash ring; reduce num_traces to something near actual concurrent traces; shorten decision_wait for the fast policies while keeping a long wait only for the slow-endpoint policy (where supported); and enable queue-size-bound backpressure so the gateway sheds load rather than OOMs. Never solve this by turning tail sampling off - you would be back to 1.8% arbitrary sampling with no error coverage.