Auto-Instrumentation Misses the Async Path
An order service is a Java Spring Boot app using the OTel Java agent (auto-instrumentation). HTTP entry spans and JDBC calls appear correctly, and the service calls a downstream inventory service over gRPC - that span is also present. But the service also does work on a @Async thread pool and a ScheduledExecutorService: spans created inside those async tasks appear as separate root traces with no parent, and the trace shows a gap where the async work should be. The team's only fix so far is to add @WithSpan on the async methods, which creates spans but still no parent linkage. They are considering turning off async processing entirely to 'fix tracing'.
1. Explain why `@WithSpan` produces orphan spans here and why the parent is lost across the thread boundary. What is the minimal correct fix?
Read the worked answer
Why the parent is lost: OTel context is stored in a thread-local (Context/ContextStorage). The agent creates the parent span on the request thread. When the task is submitted to the @Async pool or the ScheduledExecutorService, the submitting thread's context is not automatically captured and re-attached on the worker thread. The worker starts with an empty context, so @WithSpan (or any span started there) has no current parent → it becomes a root span. This is the classic context-propagation-across-threads gap.
Why @WithSpan alone doesn't help: it starts a span using whatever context is current on that thread. On a fresh pool thread that context is empty, so the span is a root. @WithSpan fixes span creation, not context hand-off.
Minimal correct fix: propagate context at the hand-off. Options, in order of preference:
- Instrumented executor wrappers - the OTel Java agent instruments
Executor,ExecutorService, andRunnable/Callablesubmission for common pools, but only if the task is submitted through an instrumented path. For Spring@Async, configure theTaskExecutorso it is wrapped (the agent'sexecutorinstrumentation coversThreadPoolExecutor/ForkJoinPool); verify the pool type is supported. - Manual wrapping - wrap submitted tasks with
Context.current().wrap(runnable)(orContext.taskWrapping(...)) at submission time so the captured context is restored on the worker thread. This works for any pool, includingScheduledExecutorService. Context.wraponCallablefor scheduled tasks; forScheduledExecutorServicespecifically, wrap at schedule time so the context is captured then.
Anti-pattern to reject: disabling async processing to 'fix' tracing is trading a real performance/correctness property for observability - never the right call.
2. After fixing propagation, the async spans now nest correctly, but the trace shows the async work completing *after* the HTTP response returns. The team wants the trace to 'wait' for async work. Should they? What is the correct modeling?
Read the worked answer
No - do not make the request trace wait for async work. The HTTP response genuinely returns before the async task completes; that is the real behavior. Forcing the trace to wait would either (a) block the request thread (a latency regression and a lie about the system), or (b) fake the end time, corrupting duration metrics.
Correct modeling - two valid patterns, pick by intent:
- Link, don't parent, for fire-and-forget work. If the async task is truly independent (background cleanup, email send), start it as a new trace and attach a Span Link to the originating span. This preserves causality without implying the request waited. The request trace ends at the response; the async trace is queryable via the link.
- Parent the span but let the trace end naturally. If the async work is part of the logical operation and you want it in the same trace, parent it (as fixed) and accept that the trace's end time is the async completion. This is fine only if the async work is genuinely part of the request's critical path and you understand the trace duration then reflects it.
Key distinction to articulate: parent-child means 'this is a child operation of that parent'; a link means 'related to, but not a child'. Using parent for fire-and-forget creates traces whose duration is dominated by unrelated background work, which distorts latency SLOs and confuses tail sampling (the trace looks long).
Recommendation: for the ScheduledExecutorService (periodic jobs, no request), use links to the triggering span or no link at all if there is no trigger. For @Async invoked from a request and awaited logically, parent it. Document the choice per call site; don't apply one rule blindly.