Tail sampling and a broken Kafka trace between checkout and inventory
Checkout is a Java Spring Boot service on three Kubernetes pods calling a Node.js cart service and publishing to Kafka; a Python inventory worker consumes the topic. All services use Elastic APM agents 8.x and report to a Fleet-managed APM Server. To catch rare failures, transaction_sample_rate is 1.0 everywhere, and the APM Server runs at 85-95 percent CPU during peak, with apm-* indices growing 700 GB per week. The service map shows checkout and cart connected, but inventory is a separate island with no upstream. A trace search for a checkout order ID returns the HTTP transaction and the Kafka produce span, but nothing from the consumer. Application logs contain trace.id only for HTTP requests; the worker logs a message ID instead. The team cannot reduce visibility into errors and wants to keep full traces for failed checkout transactions.
1. Compare head-based and tail-based sampling for this workload and recommend a configuration that keeps all failed checkout traces.
Read the worked answer
Head-based sampling decides at the root and propagates that decision, so it is cheap and consistent but cannot know whether the transaction will fail. At a 10 percent rate you lose 90 percent of failures, which violates the requirement. Raising the rate to keep failures brings back the volume and APM Server load.
Tail-based sampling buffers a trace until it completes and then decides using the whole trace. In the APM integration enable tail sampling and define policies: sample rate 1.0 for trace.outcome: failure, a moderate rate such as 0.2 for trace.name matching the checkout transaction, and a low default such as 0.05. Keep agent transaction_sample_rate at 1.0, otherwise the APM Server never sees the traces it is supposed to evaluate.
Tail sampling requires every span of a trace to reach the same APM Server instance. With multiple APM Server pods, put a load balancer in front using trace-ID affinity, or run a dedicated sampling tier. It also needs memory for in-flight traces and a timeout for traces that never complete. Size the APM Server for the full unsampled ingest before enabling it, and monitor dropped or timed-out trace groups.
2. Why is inventory missing from the trace and the service map, and how would you fix context propagation across Kafka?
Read the worked answer
HTTP propagation stops at the Kafka boundary. The produce side may write trace headers into the Kafka record, but the Python consumer must extract them and start its transaction with that remote parent. If it starts a new root transaction, the consumer has a new trace ID, no parent-child edge exists, and the service map shows inventory as an island. The produce span without a matching consumer span is the signature of this break.
Inspect the record headers with a Kafka console consumer configured to print headers, looking for traceparent or elastic-apm-traceparent. On the producer, inject the current trace context into the record headers. On the Python consumer, extract the header, start a transaction of type messaging with that trace parent, and set the destination resource for the downstream call so the service map can name the edge. Also set service.name consistently and avoid overwriting it from a generic framework name.
After the fix, a trace search should return one trace ID spanning checkout, cart, the produce span and the inventory consumer transaction. Test with a single order in staging before enabling tail sampling, so unsampled debugging is still possible.
3. How would you make the inventory worker logs correlate with APM traces, and why is the message ID insufficient?
Read the worked answer
APM log correlation depends on ECS fields, primarily trace.id and optionally transaction.id and span.id, plus service.name. The message ID is a business identifier; Kibana cannot join it to a trace unless you build a custom lookup, and it does not help when the trace ID is missing from the log line.
Add a logging filter or MDC integration to the Python worker that reads the active transaction and injects trace.id and transaction.id into every structured log record. Do this after the Kafka context fix, otherwise the worker has no shared trace ID to log. Ensure the logs are ingested as ECS documents with trace.id mapped as keyword and service.name set to the same value the APM agent reports. Then the APM UI log link and a Discover query by trace ID both work.
Keep the message ID as a separate field for business search, and add it to the trace as a label or custom span attribute only if it is low cardinality. Avoid putting high-cardinality IDs into metric dimensions; that is a storage problem, not a correlation fix.