Incomplete Checkout Traces Through Kafka and a Sampling Budget Constraint
Tailspin Toys runs 30 microservices on Kubernetes: Java, Node.js, and .NET, plus Kafka for asynchronous order events. Instana shows only about 40% of expected checkout traces. The service map has a broken edge between order-service (Java) and payment-service (.NET) when messages pass through Kafka; synchronous HTTP calls are traced correctly. The Java agent configuration sets a 10% trace sampling rate for order-service to control cost. A legacy .NET payment consumer cannot be redeployed for two weeks, and the team needs visibility now. The OpenTelemetry Collector is already in the cluster but is not used for these services. You must improve trace completeness without exceeding the current ingest budget by more than 20%. Peak checkout volume is 300 requests per minute; each trace has about 12 spans.
1. Why are only 40% of checkout traces visible, and how would you change sampling without exceeding the ingest budget by more than 20%?
Read the worked answer
Two causes are likely mixed: head-based sampling at 10% on order-service, and missing context propagation through Kafka. Sampling explains a large drop but not a clean service-map break; a broken edge usually means the consumer starts a new trace because the Kafka message lacks trace headers. I would first confirm in trace search whether payment-service spans exist as separate short traces rather than as children of order-service.
For sampling, do not raise all 30 services to 100%. At 300 requests/min and 12 spans, 100% is about 3,600 spans/min for that path versus 360 at 10%. Raising only the checkout entry service and the Kafka consumer to 100% concentrates the increase; keep other services at 10% and rely on adaptive sampling to retain errors and slow traces. If the 20% budget is still exceeded, use tail-based sampling at the OpenTelemetry Collector for checkout, keeping all errors and a lower percentage of successes. Sample at the trace entry point so the decision propagates, not independently per service.
2. How do you fix the broken Kafka edge between Java and .NET without redeploying the .NET consumer for two weeks?
Read the worked answer
A true distributed trace edge requires the producer to inject trace context into Kafka headers and the consumer to extract it. Verify the Java producer is actually injecting headers; if it is not, enabling Kafka client instrumentation there is low-risk and may restore the edge if the .NET agent already extracts. If the legacy .NET consumer does not extract headers, no producer change creates a child span; the honest short-term answer is that full trace continuity requires a consumer change or a redeploy with instrumentation that supports Kafka.
For the two-week gap, do not fake an edge. Use service-level Kafka consumer lag, error metrics, and log correlation to cover payment-service, and schedule the .NET redeploy. If the OpenTelemetry Collector can be placed in the message path, it can help with sampling/export, but it cannot invent extraction inside an uninstrumented consumer. The constraint changes the answer: partial visibility now, complete trace later.
3. How would you verify trace completeness and measure ingest impact after the change?
Read the worked answer
Use trace search filtered to checkout and count traces per minute against the 300 rpm baseline; verify payment-service spans are children of the same trace ID across Kafka, not separate traces. Inspect the service map for the order→payment edge and confirm it is not broken. Compare span ingest before/after and against the 20% budget, broken down by service so one noisy service does not hide the checkout increase. Verify error traces are retained at 100% if adaptive or tail sampling is used.
If trace count rises but the Kafka edge is still broken, the sampling fix worked and the propagation fix did not; treat those as separate success criteria.