Distributed Tracing: Traces and Spans
Distributed tracing, traces and spans
This domain tests whether you can read a trace correctly: identifier formats, span fields and kinds, context propagation, and how to find the critical path.
Anatomy
A trace is a directed acyclic graph of spans sharing a trace_id - 16 bytes, written as 32 hex characters. Each span has an 8-byte span_id (16 hex chars) and a parent_span_id, empty only for the root. A span carries a name, kind, start and end timestamps, attributes, events, status and optional links to other traces.
Span kinds: server, client, producer, consumer, internal. Client/server pairs let the backend compute network time between services.
Propagation
W3C Trace Context is the standard: a traceparent header of the form 00-<trace-id>-<parent-id>-<flags> plus optional tracestate, plus baggage for business context such as tenant or user tier. Every hop - gateway, queue, worker - must forward it, or the trace fractures into unrelated roots.
Finding the slow part
Self-time = span duration minus time attributed to children. The critical path is the chain of spans that determines total latency; optimising anything off it changes nothing.
Worked example
The API gateway root span is 800 ms. Children: auth client 50 ms, orders client 600 ms, currency client 100 ms. The orders server span is also 600 ms, and its child database query is 550 ms. Critical path: gateway → orders → database, with only ~50 ms of real application work. The team adds an index to the query rather than scaling the orders service. Reasoning: self-time, not total duration, tells you where the time is actually spent.
Traps that cost marks
- Reading a span's duration as "time spent in that service"; queueing and instrumentation overhead distort it.
- Missing propagation headers, producing many single-span traces.
- Clock skew between hosts, so a child appears to start before its parent; sync with NTP and prefer parent-reported timings.
- Spans that are started but never ended, inflating traces.
- Putting unbounded values (user IDs, raw SQL) into span names.