The trace that stops at the queue
A retail platform traces order placement across four services: api-gateway (Go) calls orders (Java, Spring Boot), which publishes to an SQS queue called order-events; fulfillment-worker (Python) consumes those messages and calls inventory (Java). Volume is roughly 1.4 million order requests per day. The APM service map renders as two disconnected clusters - api-gateway to orders, and fulfillment-worker to inventory - with no edge between orders and the worker. The worker's service page reports about 38,000 requests per day, matching SQS message volume, but every one of its traces is a root span with no parent. The orders service's p99 for POST /orders sits at 210 ms, and the team's customer-facing latency SLO is built directly on that number. Support tickets, meanwhile, consistently describe waits of three to four seconds after checkout. Head-based sampling is set to 100% for errors and 10% for everything else, and no error rate is elevated anywhere.
1. Explain why the service map splits into two disconnected graphs, and state precisely what the 210 ms p99 does and does not measure. Be specific about the mechanism that is missing.
Read the worked answer
The missing mechanism is distributed context propagation across an asynchronous boundary. The Java producer calls SQS SendMessage without injecting the active span context into the message, and the Python consumer starts its work with no parent context. dd-trace therefore mints a brand-new trace ID in fulfillment-worker, and its spans become roots - which is exactly why 38,000 worker requests per day match SQS message volume but have no parent. Nothing is broken in either tracer; the trace simply ends at the publish call and a second, unrelated trace begins at the consumer.
Consequences to reason through:
- The service map has no edge, because edges are inferred from parent/child span relationships, not from message volume.
- The 210 ms p99 covers only the synchronous portion: gateway → orders →
SendMessageacknowledgment. Queue residency time, consumer scheduling delay, and the inventory call are all outside it. That is the three-to-four seconds in the tickets. - Sampling decisions are also re-made at the consumer. The same logical order can be sampled in at the producer and out at the consumer, so per-trace funnel counts (orders created vs fulfilled) are not comparable across the boundary.
- Error correlation across the hop is impossible: a worker failure cannot be tied to the order that caused it, so the 0% worker error correlation in the orders view is an artifact, not a clean bill of health.
2. Propose the instrumentation changes required to produce one end-to-end latency number for order placement. Include how queue time must be captured, and why it should not simply be folded into the parent span's duration.
Read the worked answer
Producer (Java). Before SendMessage, take the active SpanContext and inject it as a text-map carrier into SQS message attributes - trace ID, parent span ID, sampling priority, origin. SQS attributes are typed strings, so keep keys lowercase and stable, and inject after the span starts but before the send so the parent ID is valid.
Consumer (Python). Read those attributes, reconstruct a SpanContext, and start the consume span with child_of=ctx. Tag it as a consumer operation and set messaging semantics (system, destination, message ID). Once this is in place, the worker's traces join the gateway's trace and the service map edge appears.
Queue time. No span covers residency in the queue, so it must be measured explicitly: read SentTimestamp and ApproximateFirstReceiveTimestamp from the message and emit a order.queue.delay gauge, or open a synthetic span spanning publish-to-receive. Folding minutes-long queue waits into the parent trace is a bad idea: a single backlog incident would distort every percentile on the orders service and make the SLO un-actionable, and the synthetic span's duration would be attributed to a service that did not cause it.
Then define the SLO carefully. The end-to-end number is the duration of the gateway entry span, which now closes after inventory responds. Keep queue delay as a separate SLI with its own threshold, since it has a different owner and a different remediation.
3. The team wants to raise head-based sampling to 100% so they can 'see every slow order'. Evaluate that proposal against what sampling actually controls.
Read the worked answer
It solves the wrong problem. Even at 100% head sampling, the orders p99 would still read 210 ms, because queue time is not inside any span - the missing seconds are structural, not statistical. Raising sampling changes which traces are available for search, not what is in them.
The cost side matters too. Moving application-side sampling to 100% multiplies the span volume shipped to the Agent and, depending on the ingestion controls, the volume retained. With roughly 1.4M orders per day and a multi-service fan-out, that is a meaningful and unnecessary bill.
A better sequencing:
- Fix propagation first, so the trace actually contains the wait.
- Remove application-side head sampling for this service so the Agent sees all traffic and trace metrics (hits, errors, duration, apdex) are computed from the full population rather than extrapolated from a 10% slice.
- Use retention filters to keep a representative set of slow and errored traces for investigation. Retention filters govern which traces are stored and searchable; they do not change the metrics, so they are the right lever for cost.
- Keep the error-biased sampling rule only if the Agent is not already computing error metrics from full traffic.
If the goal is 'see every slow order', the correct tool is a retention filter on latency above a threshold plus full metrics - not blanket head sampling.