OTCA: Baggage Survives HTTP but Vanishes at the Queue

Baggage Survives HTTP but Vanishes at the Queue

A checkout service calls a payment service over gRPC (OTel auto-instrumented, W3C traceparent/tracestate propagators configured), then publishes to a Kafka topic consumed by a fraud service. The team uses Baggage to carry tenant.id and experiment.bucket end-to-end so downstream spans and logs can be sliced by tenant. Traces from HTTP/gRPC calls show tenant tags correctly. But every span and log produced by the fraud service has no tenant.id, and the Kafka consumer's spans appear as new root traces (no parent). The producer uses the Java Kafka client with the OTel Kafka instrumentation; the consumer uses a plain KafkaConsumer with manual poll() in a loop, and the team says 'Kafka isn't HTTP so there's nothing to propagate'.

1. Explain why both the trace context and the baggage are lost at the Kafka boundary, and what must be configured on producer and consumer to fix each.

Read the worked answer

Why both are lost: propagation is a transport-level contract. OTel propagators (W3C TraceContext, Baggage) only inject/extract when an instrumented client library tells it to. The HTTP/gRPC clients are auto-instrumented, so traceparent, tracestate, and baggage headers are written and read. The Kafka producer, if the OTel Kafka instrumentation is active, will inject context into record headers - but only if the instrumentation is actually on the classpath and the producer is built through the instrumented path. The consumer is a plain KafkaConsumer with no instrumentation, so nothing extracts headers → new root trace, and baggage is never read → no tenant.id.

Fix - trace context: ensure the OTel Kafka instrumentation is applied to both producer and consumer (agent, or the OTel Kafka client wrapper). The instrumentation injects traceparent/tracestate/baggage into Kafka record headers on produce and extracts them on consume, creating the CONSUMER span as a child of the producer span.

Fix - baggage: baggage is carried in the same headers only if the baggage propagator is registered in the global propagators on both sides (e.g. OTEL_PROPAGATORS=tracecontext,baggage). A common failure: the propagator list is set on the producer service but not on the consumer, so even with instrumentation the baggage is extracted as empty. Verify with OTEL_PROPAGATORS env on both deployments.

2. Assume the consumer cannot be re-instrumented for a release cycle. Describe a stopgap that recovers tenant attribution for logs and metrics without full trace linkage, and state its limitations.

Read the worked answer

Stopgap: carry tenant as a Kafka record header (or message key) and read it manually.

Producer: add tenant.id and experiment.bucket as explicit Kafka headers alongside (or instead of) relying on baggage.
Consumer: read record.headers() in the poll loop and put the values into the MDC / logging context and into metric attributes on the consumer's own metrics.
For logs: set a logging context (MDC in Java) so log lines carry tenant.id; the OTel logging bridge will attach it as log attributes.
For metrics: add tenant.id as an attribute on any consumer-side counter/histogram you emit.

Limitations to state:

No trace linkage - the consumer span is still a root; you get attribution but not end-to-end trace continuity. Latency between producer and consumer cannot be reconstructed from traces.
Cardinality risk - tenant.id as a metric attribute is high-cardinality and will blow up series counts. Prefer keeping tenant in logs/traces and, if needed, a bounded dimension (e.g. tenant tier) in metrics.
Baggage semantics lost - manual headers don't participate in OTel baggage propagation, so any downstream service expecting baggage won't see it; you must re-inject manually.
Drift risk - the manual path is easy to forget when new topics are added; document it and remove once instrumentation lands.

This is a legitimate interim mitigation, not a substitute for the instrumentation fix.