Datadog APM & Distributed Tracing Fundamentals: The 3.6 second gap

The 3.6 second gap

checkout is a Java service on 12 Kubernetes pods, each limited to 2 vCPU with a 1.5 GB heap, serving about 40 requests per second. Roughly every nine minutes, p99 latency rises from 900 ms to 4.2 s for about 90 seconds and then recovers. Trace waterfalls show a servlet.request span of 4.1 s with the first child database span starting 3.6 s after the parent began; every child span completes in under 20 ms and downstream services show no latency change. Gateway logs record 504s during the spikes, and the service's own error rate stays near 0.1%. Runtime metrics show an old-generation GC pause of roughly 600 ms on the same cadence, container CPU throttling at 22%, and the http-nio thread pool at 200 of 200 active threads during the spikes. The service also runs a cache refresh job every nine minutes on the same pods. Nobody has reproduced it in staging.

1. Given a 3.6 s gap between the servlet span start and the first child span, enumerate the candidate explanations and state which evidence would distinguish them.

Read the worked answer

Interpret the gap first: it is inside the request-handling span, after the request has been accepted and dispatched to the application. That rules out load-balancer queuing and, largely, the accept queue - a request waiting for an accept thread would not yet have a servlet span. So the thread that owns this span existed but did not make progress for 3.6 s.

Candidates and discriminating evidence:

CPU throttling. The cgroup quota is exhausted and threads are descheduled in long slices. Evidence: per-second container.cpu.throttled for that pod during the spike, correlated to the second; the gap should cluster on pods with the highest throttling, and 22% average throttling over nine minutes usually masks much sharper per-second peaks.
Stop-the-world GC. Old-gen pause of 600 ms does not cover 3.6 s alone - look for several consecutive pauses, or a full GC plus a long reference-processing phase. Evidence: jvm.gc.pause timestamps aligned to the gap on that pod.
Blocking wait inside the app - a lock held by the refresh job, or an uninstrumented call (DNS, connection pool checkout) that produces no child span. Evidence: the gap is self time on the parent, and it correlates with the refresh job's execution window; threading it through the profiler shows the parked stack.
Cross-pod simultaneity test. GC and throttling are per-pod. If all 12 pods show the gap at the same wall-clock second, the cause is shared - the refresh job running everywhere on a synchronized schedule, or a shared dependency - not individual pod pressure.

The nine-minute cadence matching the refresh job is the strongest lead, but it could be coincidence: the job might cause memory pressure that triggers GC, in which case the job is upstream of both signals.

2. What would you change first, and how would you verify the fix without waiting through several nine-minute cycles?

Read the worked answer

First, close the observability gap - do not resize yet. The single most informative missing artifact is a span around the cache refresh job and around any lock or connection-pool acquisition in the request path. Without it, every hypothesis stays circumstantial, and the next resize may be another guess. Then, on one canary pod, pull the Continuous Profiler during a spike and compare wall-clock time against CPU time in the blocked frames: a method burning wall time with almost no CPU is parked or descheduled; a method burning CPU with the cgroup throttled is starvation.

Then act, in order of expected value:

If throttling is confirmed per-second: raise the CPU limit and set requests equal to limits for Guaranteed QoS, which also reduces eviction risk. 2 vCPU for 40 rps with a 200-thread pool is thin.
If GC dominates: raise the heap and reduce allocation rate; the sawtooth filling 1.5 GB every nine minutes means the refresh job's working set is being promoted into old gen.
Move the refresh job off the request-serving pods, or stagger and jitter it, so it stops competing for the same quota.

Verification without waiting. Reproduce deterministically: staging almost certainly has larger limits and fewer pods, which is why it does not show up. Set a staging pod to the same limits, run a 40 rps load, and start the refresh job on a short cycle. Watch container.cpu.throttled and jvm.gc.pause during the injected spike, confirm the 3.6 s gap reproduces, then apply the change to the canary pod and compare p99 across three nine-minute cycles before rolling out. Add a monitor on per-second throttling so the next occurrence is caught at the cause, not at the gateway's 504s.