Kafka workers producing orphan traces and unknown_service

The trace breaks at the Kafka boundary, in both directions. HTTP is auto-propagated because both sides implement the W3C headers over HTTP; Kafka messages carry headers only if something explicitly injects them. Two failures are likely present simultaneously.

Producer side (Java). The Java agent's Kafka instrumentation injects traceparent into record headers, but only if it is enabled and unobstructed. Check, in order: OTEL_INSTRUMENTATION_KAFKA_ENABLED is not false; no OTEL_JAVAAGENT_EXCLUDE/disabled instrumentation list covers kafka-clients; there is no custom ProducerInterceptor or wrapper that constructs a fresh ProducerRecord (which drops the headers the agent added); and the exporter/sampler are not set such that spans are created but non-recording. Confirm by inspecting actual record headers on a test topic - that is the fastest ground truth.

Consumer side (Go). The SDK does not know Kafka exists. The worker must extract context from the record headers and use it as the parent. Practically: build a propagation.TextMapCarrier backed by the []kafka.Header slice (Get returns string(h.Value), Set appends), then otel.GetTextMapPropagator().Extract(ctx, carrier) and start the consumer span with that context, SpanKindConsumer, and messaging semantic-convention attributes (messaging.system=kafka, messaging.destination.name, messaging.operation=process).

Batch consumption. Do not parent every message in a batch to one span - that misrepresents causality. Extract each message's context and attach them as span links to a single batch-process span, or create one consumer span per message with the extracted parent and a parent batch span. Either is defensible; links are usually cleaner for fan-in.

Also check goroutines. If the worker hands messages to a goroutine pool without carrying context.Context, the span is created from context.Background() and silently roots a new trace. That alone can explain a chunk of the 70%.

With correct propagation, parentbased_traceidratio at 0.1 makes one decision at the root and propagates the sampled flag. You get 10% of traces whole: every span or none. That is exactly what the sampler is for.

With broken propagation, each service becomes its own root and samples independently. The probability that both the Java span and the Go span are sampled is 0.1 × 0.1 = 1%, and even when both are kept they appear as two unrelated traces. So the fragmentation you see is not just cosmetic - it silently cuts effective trace completeness to about a tenth of the configured rate and makes end-to-end latency analysis impossible for 99% of requests.

Should you keep 0.1? No, not at this volume. 3,000 orders/min × 30 spans = 90,000 spans/min ≈ 130M spans/day at 100%. That is modest. Head-sample at 100% upstream, fix propagation with full data visible, then move the real sampling decision to the Collector gateway you already run, using tail sampling: keep 100% of traces with errors and 100% above a latency threshold, and take a small probabilistic slice of the rest. Head sampling cannot know an outcome, so it will always discard some of the traces you most need.

Sequencing matters: do not enable tail sampling before propagation is fixed. Tail sampling on fragmented traces is a very efficient way to build a large collection of useless partial traces. Fix the carrier, verify with header inspection, then change sampling.

unknown_service:go means service.name was never set on the Go SDK. The SDK falls back to that placeholder, which then becomes the service name in every backend - collapsing the worker into a single meaningless service and breaking the service map and any RED metrics built on service.name.

Set resource attributes at the SDK, not at the Collector. The Collector can add attributes, but it cannot know which build produced a span, so service.version must come from the application. Concretely:

• Go: resource.NewWithAttributes (or resource.NewSchemaless) merged into the TracerProvider with service.name, service.version, deployment.environment.name, and ideally service.instance.id. service.version should be the git SHA injected at build time via -ldflags, not a hand-maintained string.
• Java: OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES=service.version=<sha>,deployment.environment.name=prod. The agent also exposes OTEL_RESOURCE_ATTRIBUTES merging behaviour that is worth verifying - check the actual resource emitted on a span rather than assuming.
• Standardise the attribute keys across all 60 services before you need them.

Migration risks:

1. Renaming service.name is a breaking change for every dashboard, alert, and recording rule that references it. Pick the final name once, announce it, and update queries in the same window - or you will have a silent gap exactly when you are trying to measure the fix.
2. High-cardinality traps. service.version is bounded per deploy and fine; do not let anyone add commit-free-form or hostname noise to it. service.instance.id is high cardinality by definition - keep it out of metric labels and use it on spans only.
3. Environment as a resource attribute vs a label. If deployment.environment.name becomes a metric label, cardinality doubles or triples. Ensure it is applied deliberately, not by a Collector rule that fans out every environment.