Missing Database Spans in Checkout Traces

Root spans prove the agent is loaded and the entry point is instrumented; missing child spans narrow the problem to downstream instrumentation, propagation, or filtering.

Agent-side checks:

• Confirm the JDBC and HTTP client instrumentation modules are enabled in the agent configuration, not only the servlet/entry-point module.
• Check for exclude rules or classloader conflicts: some application servers and connection pools load JDBC drivers in a way the agent does not instrument by default.
• Confirm the agent version supports the application's JDBC driver and HTTP client library versions.
• Check agent logs for instrumentation errors, especially around HikariCP, ATP JDBC, and async executors.
• Verify context propagation across thread pools; if the DB call runs on a different executor without propagation, the child span may be detached or absent.

APM-side checks:

• Look for a span filter or sampling rule that drops non-root spans; sampling normally drops whole traces, but a filter can remove specific span types.
• Confirm the trace query is not filtering to a service or span type that hides children.
• Check the service map and span counts by type; if JDBC spans exist for other operations, the issue is operation-specific.

Do not change application code until agent logs and span-type counts show whether the spans are uninstrumented, filtered, or detached.

Start with Trace Explorer filtered to the checkout service and /checkout, sorted by duration. Inspect the slowest traces and the call tree: if JDBC spans are missing, use the root span's self-time and any application log timestamps to split the request into phases. A slow root with low self-time and a gap before response suggests a downstream call that is not instrumented; a high self-time points to application code or contention.

Use APM metrics for the service and operation to confirm the p95 trend, throughput, and error rate. Compare the p95 spike with ATP metrics such as CPU, storage, and wait events if available. If database metrics are normal and application logs show durationMs concentrated in a repository method, the database is less likely the cause.

Correlate by traceId: search Logging for the same trace ID and request ID. If the logs show a 2.6 s repository call but no JDBC span, the missing span is an instrumentation gap, not proof the database is fast. If the logs show a fast repository call and the root self-time is high, focus on application code.

Evidence that would change the conclusion: JDBC spans appear after enabling instrumentation and show most of the 2.8 s; then database or connection-pool work is the target. If enabling instrumentation shows the DB span is short, the original latency is above the database layer.

100% capture increases APM ingest volume roughly tenfold from the current 10% baseline, which raises cost and agent overhead and can create backpressure on the upload path. It also makes trace searches noisier and does not automatically give better answers if the missing child spans are an instrumentation problem; you would just collect more incomplete traces.

Recommend instead:

• Fix instrumentation first and confirm JDBC/HTTP spans appear in sampled traces. Sampling is not the reason child spans are absent in a trace that was collected.
• Raise sampling only for the checkout service or operation, for example to 25-50%, and keep lower rates for high-volume health checks.
• Keep error and slow-request visibility through APM metrics, synthetic checks, and log-based alerts. If the platform supports span filtering, use it to retain errors and high-latency traces rather than all traffic.
• Use Logging correlation with traceId for the requests that are sampled; do not assume every log line must have a trace.

Validate by comparing p95 and error rate from APM metrics at 10% versus the higher rate; if the metrics are stable and the instrumentation gap is closed, 100% is not justified.