APM Fundamentals: Traces, Spans, Services, and Resources
APM fundamentals
This domain tests your grasp of the data model behind Datadog APM: what a trace, span, service, and resource are, how tracers and the Datadog Agent move that data, and which metrics Datadog derives from it. Expect questions that ask you to choose the correct concept rather than configure one specific library.
The four nouns you must keep straight
- Service: a named unit of deployable code, set by
DD_SERVICEor the tracer default. It is the grouping key for the Service Map, Service Catalog, and most APM metrics. - Resource: the specific operation within a service, such as
GET /orders/{id}orSELECT orders. Resource names drive endpoint-level latency and error views. - Trace: the complete distributed call tree for one request, identified by a 64-bit trace ID.
- Span: one timed operation in that tree. Each span carries a span ID, parent ID, service, resource, type (
web,db,cache,custom), start time, duration, and tags.
A trace is assembled when context propagates between services. Datadog uses x-datadog-trace-id, x-datadog-parent-id, and x-datadog-sampling-priority headers, and also supports W3C traceparent. The receiving tracer creates a child span using the incoming trace ID and parent ID, so one trace ID spans every service.
Where the numbers come from
The tracer emits spans to the Datadog Agent over a local endpoint, by default port 8126. The Agent forwards them to Datadog. From those spans Datadog computes trace metrics: requests or hits, errors, latency percentiles such as p50, p75, p90, p95, and p99, plus Apdex when configured. These metrics are aggregated and are not the same as indexed spans, which are individual spans you can search.
Worked example
A request hits frontend, which calls orders, which queries Postgres. You see one trace with a root web span in frontend, a child web span in orders, and a db span for the query. The trace ID is identical across all three; parent IDs form the chain. The orders span duration includes database time, so the flame graph shows where latency is concentrated. If the database span is 1,800 ms of a 2,000 ms request, the fix belongs in the query, not the frontend.
Traps
Assuming one service equals one host; mixing up resource and service when naming spans; believing trace metrics are computed only from indexed spans; and assuming a missing trace always means a broken tracer when head-based sampling may simply have dropped it.