Datadog APM & Distributed Tracing Fundamentals flashcards

32 cards. Read the term and the definition here. The deck at the top of this page is optional.

Span

A single timed unit of work within a trace, such as one HTTP call or database query. It records service, resource, operation, duration, status, and parent span ID.

Trace

The complete record of one request's journey across services, built from spans arranged in a parent-child tree. Its total duration equals the root span's duration.

Service vs resource

Service is the logical application component emitting spans; resource is the specific endpoint or query inside it, like GET /users/:id or a SQL statement.

Critical path

The chain of nested child spans whose durations account for the largest share of total request latency. The widest span on this path is the primary optimization target.

Distributed context propagation

Passing trace ID and parent span ID across process and network boundaries using headers, so spans emitted by different services join into one trace.

Head-based sampling

A sampling decision made at trace start, before the full trace exists. Datadog's Agent computes per-service rates to reach a configured traces-per-second target.

Trace metrics

Aggregated hits, errors, and duration computed from 100 percent of spans regardless of trace sampling, keeping dashboards and monitors statistically accurate.

Unified service tagging

The three reserved tags DD_SERVICE, DD_ENV, and DD_VERSION that identify a service, its environment, and its deployed version across all Datadog products.

DD_TRACE_AGENT_PORT default

Tracers send spans to the local Datadog Agent on port 8126 by default. Override the destination with DD_AGENT_HOST and DD_TRACE_AGENT_PORT environment variables.

Auto-instrumentation

dd-trace libraries automatically create spans for common frameworks, HTTP clients, and database drivers without code changes, giving baseline visibility on first deploy.

Manual instrumentation

Creating custom spans and tags through the tracer API when auto-instrumentation misses business logic, so those operations become searchable and measurable.

Log injection

Setting DD_LOGS_INJECTION=true adds trace_id and span_id fields to log lines, letting you pivot between a log entry and its corresponding trace.

Runtime metrics

Process-level metrics such as JVM heap, garbage collection, and thread counts collected by the tracer when DD_RUNTIME_METRICS_ENABLED=true and shown on the service page.

Admission Controller

A Kubernetes component that automatically injects the Agent host, trace port, and unified service tags into pod environment variables when enabled.

Trace Explorer query syntax

Facet-based search over spans, combining service, resource, operation, status, duration, and indexed tag filters, for example service:web status:error duration:>500ms.

Retention filters

Rules deciding which spans are indexed and kept for 15 days. Unindexed spans remain searchable only in Live Search for 15 minutes.

Service Map

A dependency graph showing upstream and downstream services with request rate, error rate, and latency per edge, highlighting failing or slow dependencies visually.

Service Catalog

An inventory of every emitting service with ownership, dependencies, and health signals, used to find uninstrumented, orphaned, or degraded services.

Error Tracking

A view that groups errored spans into issues by error type and stack trace, showing first and last occurrence plus the affected service version.

Deployment Tracking

Version-change markers overlaid on APM graphs, used to correlate a latency or error spike with the deploy that introduced it.

Trace metric query example

APM dashboards and monitors query trace metrics such as avg:trace.http.request.duration{service:web}.rollup(p95) to alert on tail latency for one service.

Percentile vs average latency

Percentile aggregations like p95 and p99 expose tail latency affecting a minority of users; averages hide it by blending slow and fast requests together.

Resource breakdown

A table on the service page ranking endpoints by request count, error rate, and latency, revealing which resource dominates the service's performance profile.

Distribution metrics

Metrics stored with a sketch structure so percentiles remain accurate when aggregated across hosts and time, which is how APM latency percentiles are computed.

Trace-to-log correlation

Clicking a span opens logs sharing the same trace_id, so you can read the exact log lines produced during that specific request.

Flame graph reading

In a flame graph, span duration appears as horizontal width; the widest child on the critical path contributes most to total request latency.

Broken trace

A trace missing parent spans, usually caused by context headers not forwarded across a proxy, message queue, or asynchronous worker boundary.

N+1 query pattern

Many short sibling database spans under one request span, indicating one query per item. Fix by batching queries or eager loading the relationship.

Span error tags

Errored spans carry error.msg, error.type, and error.stack tags, which are searchable in the Trace Explorer and drive Error Tracking issue grouping.

No traces appearing

Common causes are an incorrect DD_AGENT_HOST or a blocked port 8126. Check trace-agent logs and Agent status before changing application code.

Metrics present traces missing

When trace metrics look healthy but individual traces are absent, sampling dropped them or retention filters excluded them; raise sampling or add a filter.

Version comparison

Filtering or grouping APM graphs by the version tag to compare a new deploy against the prior release and confirm whether a regression is version-specific.