Datadog APM & Distributed Tracing Fundamentals cheat sheet

A trace is the complete record of one request's path through a distributed system; it is composed of one or more spans organized in a parent-child tree.

A span represents a single unit of work (an HTTP call, DB query, cache hit) and carries a service name, resource, operation, duration, status, and trace/span IDs.

Service is the logical application component emitting spans; resource is the specific endpoint or query within it (for example GET /users/:id or SELECT users).

Trace duration equals the root span duration; the critical path is the chain of child spans whose durations sum to the largest share of total latency.

Distributed context propagation passes trace_id and parent span_id across process and network boundaries via headers (Datadog x-datadog-* headers, or W3C traceparent).

Sampling reduces ingested volume: head-based sampling decides at trace start; Datadog Agent computes per-service rates to target a configured traces-per-second goal.

Ingested span retention is 15 minutes in the Live Search index; indexed spans (retained by retention filters) are kept for 15 days by default.

APM metrics (requests, errors, latency percentiles) are generated from 100% of spans regardless of trace sampling, so dashboards and monitors stay statistically accurate.

A service's error rate is computed as errored requests divided by total requests over the selected timeframe; latency is reported as p50/p75/p90/p95/p99 distributions.

dd-trace libraries exist for Java, Python, Node.js, Go, .NET, Ruby, PHP, and C++; they auto-instrument common frameworks, HTTP clients, and database drivers.

The tracer sends spans to the Datadog Agent over localhost:8126 by default (DD_AGENT_HOST / DD_TRACE_AGENT_PORT override), and the Agent forwards to Datadog.

DD_SERVICE, DD_ENV, and DD_VERSION are the three unified service tags; setting them enables version comparison, deployment tracking, and environment filtering.

DD_TRACE_ENABLED=false disables tracing without removing the library; DD_LOGS_INJECTION=true adds trace_id and span_id to logs for correlation.

Manual instrumentation creates spans via the tracer API (for example tracer.trace in Python/Node, @Trace or @SpanTag in Java) and must set tags for searchable context.

Custom span tags become facets when indexed; use snake_case keys and set them on the span (span.set_tag / span.SetTag) before finishing the span.

Runtime metrics (JVM heap, GC, thread count; Node event loop) are collected by the tracer when DD_RUNTIME_METRICS_ENABLED=true and appear under the service page.

The Admission Controller (Kubernetes) injects DD_AGENT_HOST, DD_TRACE_AGENT_PORT, and the unified service tags into pods automatically when enabled.

Serverless tracing requires the Datadog Lambda extension or the Datadog Forwarder; the extension sends traces directly and reduces cold-start forwarding latency.

The Service Catalog lists every emitting service with ownership, dependencies, and health; it is the entry point for finding uninstrumented or orphaned services.

Service Map renders upstream/downstream dependencies with request rate, error rate, and latency per edge; thick red edges indicate the failing dependency.

The Trace Explorer supports queries on service, resource, operation, status, duration, and any indexed span tag, e.g. service:web status:error duration:>500ms.

Retention filters decide which spans are indexed for 15 days; custom filters can target specific services, resources, or tag values for long-term search.

Trace metrics (trace.http.request.hits, .errors, .duration) power dashboards and monitors and can be grouped by service, resource, version, or custom tags.

Error Tracking groups spans with status:error into issues by error type and stack trace, showing first/last seen and affected version.

Deployment Tracking overlays version-change markers on APM graphs, letting you correlate a latency or error spike with a specific deploy.

Watchdog automatically surfaces anomalies such as elevated error rates or latency on a service without requiring a manually configured monitor.

APM dashboards use trace metrics: hits (request count), errors (errored request count), and duration (latency distribution) as the three base measures.

Latency widgets should use percentile aggregations (p95, p99), not averages, because averages hide tail latency affecting a minority of users.

The Service page combines throughput, error rate, latency percentiles, dependency map, and a resource breakdown table in one view.

Distribution metrics (DDSketch) enable accurate percentile aggregation across hosts and time; APM latency percentiles are computed this way.

Breakdown by resource shows which endpoint dominates latency; breakdown by version shows whether a deploy caused a regression.

Correlate APM with logs and RUM by trace_id: clicking a span opens linked logs and, for browser requests, the corresponding RUM view.

Monitors on APM metrics use the same query syntax as dashboards, e.g. avg:trace.http.request.duration{service:web}.rollup(p95) > 1.

Notebooks and dashboards can embed a Trace Explorer query so a saved search becomes a reusable, shareable visualization.

Start from the symptom: high error rate -> inspect status:error spans and Error Tracking; high latency -> inspect the flame graph critical path.

The flame graph shows span duration as horizontal width; the widest child on the critical path is the dominant contributor to total latency.

Broken traces (missing parent spans) usually mean context propagation failed at a boundary: check header forwarding in proxies, queues, or async workers.

N+1 query patterns appear as many short sibling DB spans under one request span; fix by batching or eager loading.

Errors marked on spans carry error.msg, error.type, and error.stack tags; these are searchable and drive Error Tracking grouping.

Compare a slow trace against a baseline trace of the same resource to spot which child span grew; use the version tag to isolate a bad deploy.

Agent connectivity issues (no traces appearing) are diagnosed with agent status and datadog-agent trace-agent logs; common causes are wrong DD_AGENT_HOST or blocked port 8126.

If trace metrics look correct but individual traces are missing, sampling or retention filters are the cause; raise the sampling rate or add a retention filter.

A trace is the complete record of one request's path through a distributed system; it is composed of one or more spans organized in a parent-child tree.

A span represents a single unit of work (an HTTP call, DB query, cache hit) and carries a service name, resource, operation, duration, status, and trace/span IDs.

Service is the logical application component emitting spans; resource is the specific endpoint or query within it (for example GET /users/:id or SELECT users).

Trace duration equals the root span duration; the critical path is the chain of child spans whose durations sum to the largest share of total latency.

Distributed context propagation passes trace_id and parent span_id across process and network boundaries via headers (Datadog x-datadog-* headers, or W3C traceparent).

Sampling reduces ingested volume: head-based sampling decides at trace start; Datadog Agent computes per-service rates to target a configured traces-per-second goal.

Ingested span retention is 15 minutes in the Live Search index; indexed spans (retained by retention filters) are kept for 15 days by default.

APM metrics (requests, errors, latency percentiles) are generated from 100% of spans regardless of trace sampling, so dashboards and monitors stay statistically accurate.

A service's error rate is computed as errored requests divided by total requests over the selected timeframe; latency is reported as p50/p75/p90/p95/p99 distributions.

dd-trace libraries exist for Java, Python, Node.js, Go, .NET, Ruby, PHP, and C++; they auto-instrument common frameworks, HTTP clients, and database drivers.

The tracer sends spans to the Datadog Agent over localhost:8126 by default (DD_AGENT_HOST / DD_TRACE_AGENT_PORT override), and the Agent forwards to Datadog.

DD_SERVICE, DD_ENV, and DD_VERSION are the three unified service tags; setting them enables version comparison, deployment tracking, and environment filtering.

DD_TRACE_ENABLED=false disables tracing without removing the library; DD_LOGS_INJECTION=true adds trace_id and span_id to logs for correlation.

Manual instrumentation creates spans via the tracer API (for example tracer.trace in Python/Node, @Trace or @SpanTag in Java) and must set tags for searchable context.

Custom span tags become facets when indexed; use snake_case keys and set them on the span (span.set_tag / span.SetTag) before finishing the span.

Runtime metrics (JVM heap, GC, thread count; Node event loop) are collected by the tracer when DD_RUNTIME_METRICS_ENABLED=true and appear under the service page.

The Admission Controller (Kubernetes) injects DD_AGENT_HOST, DD_TRACE_AGENT_PORT, and the unified service tags into pods automatically when enabled.

Serverless tracing requires the Datadog Lambda extension or the Datadog Forwarder; the extension sends traces directly and reduces cold-start forwarding latency.

The Service Catalog lists every emitting service with ownership, dependencies, and health; it is the entry point for finding uninstrumented or orphaned services.

Service Map renders upstream/downstream dependencies with request rate, error rate, and latency per edge; thick red edges indicate the failing dependency.

The Trace Explorer supports queries on service, resource, operation, status, duration, and any indexed span tag, e.g. service:web status:error duration:>500ms.

Retention filters decide which spans are indexed for 15 days; custom filters can target specific services, resources, or tag values for long-term search.

Trace metrics (trace.http.request.hits, .errors, .duration) power dashboards and monitors and can be grouped by service, resource, version, or custom tags.

Error Tracking groups spans with status:error into issues by error type and stack trace, showing first/last seen and affected version.

Deployment Tracking overlays version-change markers on APM graphs, letting you correlate a latency or error spike with a specific deploy.

Watchdog automatically surfaces anomalies such as elevated error rates or latency on a service without requiring a manually configured monitor.

APM dashboards use trace metrics: hits (request count), errors (errored request count), and duration (latency distribution) as the three base measures.

Latency widgets should use percentile aggregations (p95, p99), not averages, because averages hide tail latency affecting a minority of users.

The Service page combines throughput, error rate, latency percentiles, dependency map, and a resource breakdown table in one view.

Distribution metrics (DDSketch) enable accurate percentile aggregation across hosts and time; APM latency percentiles are computed this way.

Breakdown by resource shows which endpoint dominates latency; breakdown by version shows whether a deploy caused a regression.

Correlate APM with logs and RUM by trace_id: clicking a span opens linked logs and, for browser requests, the corresponding RUM view.

Monitors on APM metrics use the same query syntax as dashboards, e.g. avg:trace.http.request.duration{service:web}.rollup(p95) > 1.

Notebooks and dashboards can embed a Trace Explorer query so a saved search becomes a reusable, shareable visualization.

Start from the symptom: high error rate -> inspect status:error spans and Error Tracking; high latency -> inspect the flame graph critical path.

The flame graph shows span duration as horizontal width; the widest child on the critical path is the dominant contributor to total latency.

Broken traces (missing parent spans) usually mean context propagation failed at a boundary: check header forwarding in proxies, queues, or async workers.

N+1 query patterns appear as many short sibling DB spans under one request span; fix by batching or eager loading.

Errors marked on spans carry error.msg, error.type, and error.stack tags; these are searchable and drive Error Tracking grouping.

Compare a slow trace against a baseline trace of the same resource to spot which child span grew; use the version tag to isolate a bad deploy.

Agent connectivity issues (no traces appearing) are diagnosed with agent status and datadog-agent trace-agent logs; common causes are wrong DD_AGENT_HOST or blocked port 8126.

If trace metrics look correct but individual traces are missing, sampling or retention filters are the cause; raise the sampling rate or add a retention filter.

A trace is the complete record of one request's path through a distributed system; it is composed of one or more spans organized in a parent-child tree.

A span represents a single unit of work (an HTTP call, DB query, cache hit) and carries a service name, resource, operation, duration, status, and trace/span IDs.

Service is the logical application component emitting spans; resource is the specific endpoint or query within it (for example GET /users/:id or SELECT users).

Trace duration equals the root span duration; the critical path is the chain of child spans whose durations sum to the largest share of total latency.

Distributed context propagation passes trace_id and parent span_id across process and network boundaries via headers (Datadog x-datadog-* headers, or W3C traceparent).

Sampling reduces ingested volume: head-based sampling decides at trace start; Datadog Agent computes per-service rates to target a configured traces-per-second goal.

Ingested span retention is 15 minutes in the Live Search index; indexed spans (retained by retention filters) are kept for 15 days by default.

APM metrics (requests, errors, latency percentiles) are generated from 100% of spans regardless of trace sampling, so dashboards and monitors stay statistically accurate.

A service's error rate is computed as errored requests divided by total requests over the selected timeframe; latency is reported as p50/p75/p90/p95/p99 distributions.

dd-trace libraries exist for Java, Python, Node.js, Go, .NET, Ruby, PHP, and C++; they auto-instrument common frameworks, HTTP clients, and database drivers.

The tracer sends spans to the Datadog Agent over localhost:8126 by default (DD_AGENT_HOST / DD_TRACE_AGENT_PORT override), and the Agent forwards to Datadog.

DD_SERVICE, DD_ENV, and DD_VERSION are the three unified service tags; setting them enables version comparison, deployment tracking, and environment filtering.

DD_TRACE_ENABLED=false disables tracing without removing the library; DD_LOGS_INJECTION=true adds trace_id and span_id to logs for correlation.

Manual instrumentation creates spans via the tracer API (for example tracer.trace in Python/Node, @Trace or @SpanTag in Java) and must set tags for searchable context.

Custom span tags become facets when indexed; use snake_case keys and set them on the span (span.set_tag / span.SetTag) before finishing the span.

Runtime metrics (JVM heap, GC, thread count; Node event loop) are collected by the tracer when DD_RUNTIME_METRICS_ENABLED=true and appear under the service page.

The Admission Controller (Kubernetes) injects DD_AGENT_HOST, DD_TRACE_AGENT_PORT, and the unified service tags into pods automatically when enabled.

Serverless tracing requires the Datadog Lambda extension or the Datadog Forwarder; the extension sends traces directly and reduces cold-start forwarding latency.

The Service Catalog lists every emitting service with ownership, dependencies, and health; it is the entry point for finding uninstrumented or orphaned services.

Service Map renders upstream/downstream dependencies with request rate, error rate, and latency per edge; thick red edges indicate the failing dependency.

The Trace Explorer supports queries on service, resource, operation, status, duration, and any indexed span tag, e.g. service:web status:error duration:>500ms.

Retention filters decide which spans are indexed for 15 days; custom filters can target specific services, resources, or tag values for long-term search.

Trace metrics (trace.http.request.hits, .errors, .duration) power dashboards and monitors and can be grouped by service, resource, version, or custom tags.

Error Tracking groups spans with status:error into issues by error type and stack trace, showing first/last seen and affected version.

Deployment Tracking overlays version-change markers on APM graphs, letting you correlate a latency or error spike with a specific deploy.

Watchdog automatically surfaces anomalies such as elevated error rates or latency on a service without requiring a manually configured monitor.

APM dashboards use trace metrics: hits (request count), errors (errored request count), and duration (latency distribution) as the three base measures.

Latency widgets should use percentile aggregations (p95, p99), not averages, because averages hide tail latency affecting a minority of users.

The Service page combines throughput, error rate, latency percentiles, dependency map, and a resource breakdown table in one view.

Distribution metrics (DDSketch) enable accurate percentile aggregation across hosts and time; APM latency percentiles are computed this way.

Breakdown by resource shows which endpoint dominates latency; breakdown by version shows whether a deploy caused a regression.

Correlate APM with logs and RUM by trace_id: clicking a span opens linked logs and, for browser requests, the corresponding RUM view.

Monitors on APM metrics use the same query syntax as dashboards, e.g. avg:trace.http.request.duration{service:web}.rollup(p95) > 1.

Notebooks and dashboards can embed a Trace Explorer query so a saved search becomes a reusable, shareable visualization.

Start from the symptom: high error rate -> inspect status:error spans and Error Tracking; high latency -> inspect the flame graph critical path.

The flame graph shows span duration as horizontal width; the widest child on the critical path is the dominant contributor to total latency.

Broken traces (missing parent spans) usually mean context propagation failed at a boundary: check header forwarding in proxies, queues, or async workers.

N+1 query patterns appear as many short sibling DB spans under one request span; fix by batching or eager loading.

Errors marked on spans carry error.msg, error.type, and error.stack tags; these are searchable and drive Error Tracking grouping.

Compare a slow trace against a baseline trace of the same resource to spot which child span grew; use the version tag to isolate a bad deploy.

Agent connectivity issues (no traces appearing) are diagnosed with agent status and datadog-agent trace-agent logs; common causes are wrong DD_AGENT_HOST or blocked port 8126.

If trace metrics look correct but individual traces are missing, sampling or retention filters are the cause; raise the sampling rate or add a retention filter.

A trace is the complete record of one request's path through a distributed system; it is composed of one or more spans organized in a parent-child tree.

A span represents a single unit of work (an HTTP call, DB query, cache hit) and carries a service name, resource, operation, duration, status, and trace/span IDs.

Service is the logical application component emitting spans; resource is the specific endpoint or query within it (for example GET /users/:id or SELECT users).

Trace duration equals the root span duration; the critical path is the chain of child spans whose durations sum to the largest share of total latency.

Distributed context propagation passes trace_id and parent span_id across process and network boundaries via headers (Datadog x-datadog-* headers, or W3C traceparent).

Sampling reduces ingested volume: head-based sampling decides at trace start; Datadog Agent computes per-service rates to target a configured traces-per-second goal.

Ingested span retention is 15 minutes in the Live Search index; indexed spans (retained by retention filters) are kept for 15 days by default.

APM metrics (requests, errors, latency percentiles) are generated from 100% of spans regardless of trace sampling, so dashboards and monitors stay statistically accurate.

A service's error rate is computed as errored requests divided by total requests over the selected timeframe; latency is reported as p50/p75/p90/p95/p99 distributions.

dd-trace libraries exist for Java, Python, Node.js, Go, .NET, Ruby, PHP, and C++; they auto-instrument common frameworks, HTTP clients, and database drivers.

The tracer sends spans to the Datadog Agent over localhost:8126 by default (DD_AGENT_HOST / DD_TRACE_AGENT_PORT override), and the Agent forwards to Datadog.

DD_SERVICE, DD_ENV, and DD_VERSION are the three unified service tags; setting them enables version comparison, deployment tracking, and environment filtering.

DD_TRACE_ENABLED=false disables tracing without removing the library; DD_LOGS_INJECTION=true adds trace_id and span_id to logs for correlation.

Manual instrumentation creates spans via the tracer API (for example tracer.trace in Python/Node, @Trace or @SpanTag in Java) and must set tags for searchable context.

Custom span tags become facets when indexed; use snake_case keys and set them on the span (span.set_tag / span.SetTag) before finishing the span.

Runtime metrics (JVM heap, GC, thread count; Node event loop) are collected by the tracer when DD_RUNTIME_METRICS_ENABLED=true and appear under the service page.

The Admission Controller (Kubernetes) injects DD_AGENT_HOST, DD_TRACE_AGENT_PORT, and the unified service tags into pods automatically when enabled.

Serverless tracing requires the Datadog Lambda extension or the Datadog Forwarder; the extension sends traces directly and reduces cold-start forwarding latency.

The Service Catalog lists every emitting service with ownership, dependencies, and health; it is the entry point for finding uninstrumented or orphaned services.

Service Map renders upstream/downstream dependencies with request rate, error rate, and latency per edge; thick red edges indicate the failing dependency.

The Trace Explorer supports queries on service, resource, operation, status, duration, and any indexed span tag, e.g. service:web status:error duration:>500ms.

Retention filters decide which spans are indexed for 15 days; custom filters can target specific services, resources, or tag values for long-term search.

Trace metrics (trace.http.request.hits, .errors, .duration) power dashboards and monitors and can be grouped by service, resource, version, or custom tags.

Error Tracking groups spans with status:error into issues by error type and stack trace, showing first/last seen and affected version.

Deployment Tracking overlays version-change markers on APM graphs, letting you correlate a latency or error spike with a specific deploy.

Watchdog automatically surfaces anomalies such as elevated error rates or latency on a service without requiring a manually configured monitor.

APM dashboards use trace metrics: hits (request count), errors (errored request count), and duration (latency distribution) as the three base measures.

Latency widgets should use percentile aggregations (p95, p99), not averages, because averages hide tail latency affecting a minority of users.

The Service page combines throughput, error rate, latency percentiles, dependency map, and a resource breakdown table in one view.

Distribution metrics (DDSketch) enable accurate percentile aggregation across hosts and time; APM latency percentiles are computed this way.

Breakdown by resource shows which endpoint dominates latency; breakdown by version shows whether a deploy caused a regression.

Correlate APM with logs and RUM by trace_id: clicking a span opens linked logs and, for browser requests, the corresponding RUM view.

Monitors on APM metrics use the same query syntax as dashboards, e.g. avg:trace.http.request.duration{service:web}.rollup(p95) > 1.

Notebooks and dashboards can embed a Trace Explorer query so a saved search becomes a reusable, shareable visualization.

Start from the symptom: high error rate -> inspect status:error spans and Error Tracking; high latency -> inspect the flame graph critical path.

The flame graph shows span duration as horizontal width; the widest child on the critical path is the dominant contributor to total latency.

Broken traces (missing parent spans) usually mean context propagation failed at a boundary: check header forwarding in proxies, queues, or async workers.

N+1 query patterns appear as many short sibling DB spans under one request span; fix by batching or eager loading.

Errors marked on spans carry error.msg, error.type, and error.stack tags; these are searchable and drive Error Tracking grouping.

Compare a slow trace against a baseline trace of the same resource to spot which child span grew; use the version tag to isolate a bad deploy.

Agent connectivity issues (no traces appearing) are diagnosed with agent status and datadog-agent trace-agent logs; common causes are wrong DD_AGENT_HOST or blocked port 8126.

If trace metrics look correct but individual traces are missing, sampling or retention filters are the cause; raise the sampling rate or add a retention filter.

A trace is the complete record of one request's path through a distributed system; it is composed of one or more spans organized in a parent-child tree.

A span represents a single unit of work (an HTTP call, DB query, cache hit) and carries a service name, resource, operation, duration, status, and trace/span IDs.

Service is the logical application component emitting spans; resource is the specific endpoint or query within it (for example GET /users/:id or SELECT users).

Trace duration equals the root span duration; the critical path is the chain of child spans whose durations sum to the largest share of total latency.

Distributed context propagation passes trace_id and parent span_id across process and network boundaries via headers (Datadog x-datadog-* headers, or W3C traceparent).

Sampling reduces ingested volume: head-based sampling decides at trace start; Datadog Agent computes per-service rates to target a configured traces-per-second goal.

Ingested span retention is 15 minutes in the Live Search index; indexed spans (retained by retention filters) are kept for 15 days by default.

APM metrics (requests, errors, latency percentiles) are generated from 100% of spans regardless of trace sampling, so dashboards and monitors stay statistically accurate.

A service's error rate is computed as errored requests divided by total requests over the selected timeframe; latency is reported as p50/p75/p90/p95/p99 distributions.

dd-trace libraries exist for Java, Python, Node.js, Go, .NET, Ruby, PHP, and C++; they auto-instrument common frameworks, HTTP clients, and database drivers.

The tracer sends spans to the Datadog Agent over localhost:8126 by default (DD_AGENT_HOST / DD_TRACE_AGENT_PORT override), and the Agent forwards to Datadog.

DD_SERVICE, DD_ENV, and DD_VERSION are the three unified service tags; setting them enables version comparison, deployment tracking, and environment filtering.

DD_TRACE_ENABLED=false disables tracing without removing the library; DD_LOGS_INJECTION=true adds trace_id and span_id to logs for correlation.

Manual instrumentation creates spans via the tracer API (for example tracer.trace in Python/Node, @Trace or @SpanTag in Java) and must set tags for searchable context.

Custom span tags become facets when indexed; use snake_case keys and set them on the span (span.set_tag / span.SetTag) before finishing the span.

Runtime metrics (JVM heap, GC, thread count; Node event loop) are collected by the tracer when DD_RUNTIME_METRICS_ENABLED=true and appear under the service page.

The Admission Controller (Kubernetes) injects DD_AGENT_HOST, DD_TRACE_AGENT_PORT, and the unified service tags into pods automatically when enabled.

Serverless tracing requires the Datadog Lambda extension or the Datadog Forwarder; the extension sends traces directly and reduces cold-start forwarding latency.

The Service Catalog lists every emitting service with ownership, dependencies, and health; it is the entry point for finding uninstrumented or orphaned services.

Service Map renders upstream/downstream dependencies with request rate, error rate, and latency per edge; thick red edges indicate the failing dependency.

The Trace Explorer supports queries on service, resource, operation, status, duration, and any indexed span tag, e.g. service:web status:error duration:>500ms.

Retention filters decide which spans are indexed for 15 days; custom filters can target specific services, resources, or tag values for long-term search.

Trace metrics (trace.http.request.hits, .errors, .duration) power dashboards and monitors and can be grouped by service, resource, version, or custom tags.

Error Tracking groups spans with status:error into issues by error type and stack trace, showing first/last seen and affected version.

Deployment Tracking overlays version-change markers on APM graphs, letting you correlate a latency or error spike with a specific deploy.

Watchdog automatically surfaces anomalies such as elevated error rates or latency on a service without requiring a manually configured monitor.

APM dashboards use trace metrics: hits (request count), errors (errored request count), and duration (latency distribution) as the three base measures.

Latency widgets should use percentile aggregations (p95, p99), not averages, because averages hide tail latency affecting a minority of users.

The Service page combines throughput, error rate, latency percentiles, dependency map, and a resource breakdown table in one view.

Distribution metrics (DDSketch) enable accurate percentile aggregation across hosts and time; APM latency percentiles are computed this way.

Breakdown by resource shows which endpoint dominates latency; breakdown by version shows whether a deploy caused a regression.

Correlate APM with logs and RUM by trace_id: clicking a span opens linked logs and, for browser requests, the corresponding RUM view.

Monitors on APM metrics use the same query syntax as dashboards, e.g. avg:trace.http.request.duration{service:web}.rollup(p95) > 1.

Notebooks and dashboards can embed a Trace Explorer query so a saved search becomes a reusable, shareable visualization.

Start from the symptom: high error rate -> inspect status:error spans and Error Tracking; high latency -> inspect the flame graph critical path.

The flame graph shows span duration as horizontal width; the widest child on the critical path is the dominant contributor to total latency.

Broken traces (missing parent spans) usually mean context propagation failed at a boundary: check header forwarding in proxies, queues, or async workers.

N+1 query patterns appear as many short sibling DB spans under one request span; fix by batching or eager loading.

Errors marked on spans carry error.msg, error.type, and error.stack tags; these are searchable and drive Error Tracking grouping.

Compare a slow trace against a baseline trace of the same resource to spot which child span grew; use the version tag to isolate a bad deploy.

Agent connectivity issues (no traces appearing) are diagnosed with agent status and datadog-agent trace-agent logs; common causes are wrong DD_AGENT_HOST or blocked port 8126.

If trace metrics look correct but individual traces are missing, sampling or retention filters are the cause; raise the sampling rate or add a retention filter.

A trace is the complete record of one request's path through a distributed system; it is composed of one or more spans organized in a parent-child tree.

A span represents a single unit of work (an HTTP call, DB query, cache hit) and carries a service name, resource, operation, duration, status, and trace/span IDs.

Service is the logical application component emitting spans; resource is the specific endpoint or query within it (for example GET /users/:id or SELECT users).

Trace duration equals the root span duration; the critical path is the chain of child spans whose durations sum to the largest share of total latency.

Distributed context propagation passes trace_id and parent span_id across process and network boundaries via headers (Datadog x-datadog-* headers, or W3C traceparent).

Sampling reduces ingested volume: head-based sampling decides at trace start; Datadog Agent computes per-service rates to target a configured traces-per-second goal.

Ingested span retention is 15 minutes in the Live Search index; indexed spans (retained by retention filters) are kept for 15 days by default.

APM metrics (requests, errors, latency percentiles) are generated from 100% of spans regardless of trace sampling, so dashboards and monitors stay statistically accurate.

A service's error rate is computed as errored requests divided by total requests over the selected timeframe; latency is reported as p50/p75/p90/p95/p99 distributions.

dd-trace libraries exist for Java, Python, Node.js, Go, .NET, Ruby, PHP, and C++; they auto-instrument common frameworks, HTTP clients, and database drivers.

The tracer sends spans to the Datadog Agent over localhost:8126 by default (DD_AGENT_HOST / DD_TRACE_AGENT_PORT override), and the Agent forwards to Datadog.

DD_SERVICE, DD_ENV, and DD_VERSION are the three unified service tags; setting them enables version comparison, deployment tracking, and environment filtering.

DD_TRACE_ENABLED=false disables tracing without removing the library; DD_LOGS_INJECTION=true adds trace_id and span_id to logs for correlation.

Manual instrumentation creates spans via the tracer API (for example tracer.trace in Python/Node, @Trace or @SpanTag in Java) and must set tags for searchable context.

Custom span tags become facets when indexed; use snake_case keys and set them on the span (span.set_tag / span.SetTag) before finishing the span.

Runtime metrics (JVM heap, GC, thread count; Node event loop) are collected by the tracer when DD_RUNTIME_METRICS_ENABLED=true and appear under the service page.

The Admission Controller (Kubernetes) injects DD_AGENT_HOST, DD_TRACE_AGENT_PORT, and the unified service tags into pods automatically when enabled.

Serverless tracing requires the Datadog Lambda extension or the Datadog Forwarder; the extension sends traces directly and reduces cold-start forwarding latency.

The Service Catalog lists every emitting service with ownership, dependencies, and health; it is the entry point for finding uninstrumented or orphaned services.

Service Map renders upstream/downstream dependencies with request rate, error rate, and latency per edge; thick red edges indicate the failing dependency.

The Trace Explorer supports queries on service, resource, operation, status, duration, and any indexed span tag, e.g. service:web status:error duration:>500ms.

Retention filters decide which spans are indexed for 15 days; custom filters can target specific services, resources, or tag values for long-term search.

Trace metrics (trace.http.request.hits, .errors, .duration) power dashboards and monitors and can be grouped by service, resource, version, or custom tags.

Error Tracking groups spans with status:error into issues by error type and stack trace, showing first/last seen and affected version.

Deployment Tracking overlays version-change markers on APM graphs, letting you correlate a latency or error spike with a specific deploy.

Watchdog automatically surfaces anomalies such as elevated error rates or latency on a service without requiring a manually configured monitor.

APM dashboards use trace metrics: hits (request count), errors (errored request count), and duration (latency distribution) as the three base measures.

Latency widgets should use percentile aggregations (p95, p99), not averages, because averages hide tail latency affecting a minority of users.

The Service page combines throughput, error rate, latency percentiles, dependency map, and a resource breakdown table in one view.

Distribution metrics (DDSketch) enable accurate percentile aggregation across hosts and time; APM latency percentiles are computed this way.

Breakdown by resource shows which endpoint dominates latency; breakdown by version shows whether a deploy caused a regression.

Correlate APM with logs and RUM by trace_id: clicking a span opens linked logs and, for browser requests, the corresponding RUM view.

Monitors on APM metrics use the same query syntax as dashboards, e.g. avg:trace.http.request.duration{service:web}.rollup(p95) > 1.

Notebooks and dashboards can embed a Trace Explorer query so a saved search becomes a reusable, shareable visualization.

Start from the symptom: high error rate -> inspect status:error spans and Error Tracking; high latency -> inspect the flame graph critical path.

The flame graph shows span duration as horizontal width; the widest child on the critical path is the dominant contributor to total latency.

Broken traces (missing parent spans) usually mean context propagation failed at a boundary: check header forwarding in proxies, queues, or async workers.

N+1 query patterns appear as many short sibling DB spans under one request span; fix by batching or eager loading.

Errors marked on spans carry error.msg, error.type, and error.stack tags; these are searchable and drive Error Tracking grouping.

Compare a slow trace against a baseline trace of the same resource to spot which child span grew; use the version tag to isolate a bad deploy.

Agent connectivity issues (no traces appearing) are diagnosed with agent status and datadog-agent trace-agent logs; common causes are wrong DD_AGENT_HOST or blocked port 8126.

If trace metrics look correct but individual traces are missing, sampling or retention filters are the cause; raise the sampling rate or add a retention filter.

A trace is the complete record of one request's path through a distributed system; it is composed of one or more spans organized in a parent-child tree.

A span represents a single unit of work (an HTTP call, DB query, cache hit) and carries a service name, resource, operation, duration, status, and trace/span IDs.

Service is the logical application component emitting spans; resource is the specific endpoint or query within it (for example GET /users/:id or SELECT users).

Trace duration equals the root span duration; the critical path is the chain of child spans whose durations sum to the largest share of total latency.

Distributed context propagation passes trace_id and parent span_id across process and network boundaries via headers (Datadog x-datadog-* headers, or W3C traceparent).

Sampling reduces ingested volume: head-based sampling decides at trace start; Datadog Agent computes per-service rates to target a configured traces-per-second goal.

Ingested span retention is 15 minutes in the Live Search index; indexed spans (retained by retention filters) are kept for 15 days by default.

APM metrics (requests, errors, latency percentiles) are generated from 100% of spans regardless of trace sampling, so dashboards and monitors stay statistically accurate.

A service's error rate is computed as errored requests divided by total requests over the selected timeframe; latency is reported as p50/p75/p90/p95/p99 distributions.

dd-trace libraries exist for Java, Python, Node.js, Go, .NET, Ruby, PHP, and C++; they auto-instrument common frameworks, HTTP clients, and database drivers.

The tracer sends spans to the Datadog Agent over localhost:8126 by default (DD_AGENT_HOST / DD_TRACE_AGENT_PORT override), and the Agent forwards to Datadog.

DD_SERVICE, DD_ENV, and DD_VERSION are the three unified service tags; setting them enables version comparison, deployment tracking, and environment filtering.

DD_TRACE_ENABLED=false disables tracing without removing the library; DD_LOGS_INJECTION=true adds trace_id and span_id to logs for correlation.

Manual instrumentation creates spans via the tracer API (for example tracer.trace in Python/Node, @Trace or @SpanTag in Java) and must set tags for searchable context.

Custom span tags become facets when indexed; use snake_case keys and set them on the span (span.set_tag / span.SetTag) before finishing the span.

Runtime metrics (JVM heap, GC, thread count; Node event loop) are collected by the tracer when DD_RUNTIME_METRICS_ENABLED=true and appear under the service page.

The Admission Controller (Kubernetes) injects DD_AGENT_HOST, DD_TRACE_AGENT_PORT, and the unified service tags into pods automatically when enabled.

Serverless tracing requires the Datadog Lambda extension or the Datadog Forwarder; the extension sends traces directly and reduces cold-start forwarding latency.

The Service Catalog lists every emitting service with ownership, dependencies, and health; it is the entry point for finding uninstrumented or orphaned services.

Service Map renders upstream/downstream dependencies with request rate, error rate, and latency per edge; thick red edges indicate the failing dependency.

The Trace Explorer supports queries on service, resource, operation, status, duration, and any indexed span tag, e.g. service:web status:error duration:>500ms.

Retention filters decide which spans are indexed for 15 days; custom filters can target specific services, resources, or tag values for long-term search.

Trace metrics (trace.http.request.hits, .errors, .duration) power dashboards and monitors and can be grouped by service, resource, version, or custom tags.

Error Tracking groups spans with status:error into issues by error type and stack trace, showing first/last seen and affected version.

Deployment Tracking overlays version-change markers on APM graphs, letting you correlate a latency or error spike with a specific deploy.

Watchdog automatically surfaces anomalies such as elevated error rates or latency on a service without requiring a manually configured monitor.

APM dashboards use trace metrics: hits (request count), errors (errored request count), and duration (latency distribution) as the three base measures.

Latency widgets should use percentile aggregations (p95, p99), not averages, because averages hide tail latency affecting a minority of users.

The Service page combines throughput, error rate, latency percentiles, dependency map, and a resource breakdown table in one view.

Distribution metrics (DDSketch) enable accurate percentile aggregation across hosts and time; APM latency percentiles are computed this way.

Breakdown by resource shows which endpoint dominates latency; breakdown by version shows whether a deploy caused a regression.

Correlate APM with logs and RUM by trace_id: clicking a span opens linked logs and, for browser requests, the corresponding RUM view.

Monitors on APM metrics use the same query syntax as dashboards, e.g. avg:trace.http.request.duration{service:web}.rollup(p95) > 1.

Notebooks and dashboards can embed a Trace Explorer query so a saved search becomes a reusable, shareable visualization.

Start from the symptom: high error rate -> inspect status:error spans and Error Tracking; high latency -> inspect the flame graph critical path.

The flame graph shows span duration as horizontal width; the widest child on the critical path is the dominant contributor to total latency.

Broken traces (missing parent spans) usually mean context propagation failed at a boundary: check header forwarding in proxies, queues, or async workers.

N+1 query patterns appear as many short sibling DB spans under one request span; fix by batching or eager loading.

Errors marked on spans carry error.msg, error.type, and error.stack tags; these are searchable and drive Error Tracking grouping.

Compare a slow trace against a baseline trace of the same resource to spot which child span grew; use the version tag to isolate a bad deploy.

Agent connectivity issues (no traces appearing) are diagnosed with agent status and datadog-agent trace-agent logs; common causes are wrong DD_AGENT_HOST or blocked port 8126.

If trace metrics look correct but individual traces are missing, sampling or retention filters are the cause; raise the sampling rate or add a retention filter.

A trace is the complete record of one request's path through a distributed system; it is composed of one or more spans organized in a parent-child tree.

A span represents a single unit of work (an HTTP call, DB query, cache hit) and carries a service name, resource, operation, duration, status, and trace/span IDs.

Service is the logical application component emitting spans; resource is the specific endpoint or query within it (for example GET /users/:id or SELECT users).

Trace duration equals the root span duration; the critical path is the chain of child spans whose durations sum to the largest share of total latency.

Distributed context propagation passes trace_id and parent span_id across process and network boundaries via headers (Datadog x-datadog-* headers, or W3C traceparent).

Sampling reduces ingested volume: head-based sampling decides at trace start; Datadog Agent computes per-service rates to target a configured traces-per-second goal.

Ingested span retention is 15 minutes in the Live Search index; indexed spans (retained by retention filters) are kept for 15 days by default.

APM metrics (requests, errors, latency percentiles) are generated from 100% of spans regardless of trace sampling, so dashboards and monitors stay statistically accurate.

A service's error rate is computed as errored requests divided by total requests over the selected timeframe; latency is reported as p50/p75/p90/p95/p99 distributions.

dd-trace libraries exist for Java, Python, Node.js, Go, .NET, Ruby, PHP, and C++; they auto-instrument common frameworks, HTTP clients, and database drivers.

The tracer sends spans to the Datadog Agent over localhost:8126 by default (DD_AGENT_HOST / DD_TRACE_AGENT_PORT override), and the Agent forwards to Datadog.

DD_SERVICE, DD_ENV, and DD_VERSION are the three unified service tags; setting them enables version comparison, deployment tracking, and environment filtering.

DD_TRACE_ENABLED=false disables tracing without removing the library; DD_LOGS_INJECTION=true adds trace_id and span_id to logs for correlation.

Manual instrumentation creates spans via the tracer API (for example tracer.trace in Python/Node, @Trace or @SpanTag in Java) and must set tags for searchable context.

Custom span tags become facets when indexed; use snake_case keys and set them on the span (span.set_tag / span.SetTag) before finishing the span.

Runtime metrics (JVM heap, GC, thread count; Node event loop) are collected by the tracer when DD_RUNTIME_METRICS_ENABLED=true and appear under the service page.

The Admission Controller (Kubernetes) injects DD_AGENT_HOST, DD_TRACE_AGENT_PORT, and the unified service tags into pods automatically when enabled.

Serverless tracing requires the Datadog Lambda extension or the Datadog Forwarder; the extension sends traces directly and reduces cold-start forwarding latency.

The Service Catalog lists every emitting service with ownership, dependencies, and health; it is the entry point for finding uninstrumented or orphaned services.

Service Map renders upstream/downstream dependencies with request rate, error rate, and latency per edge; thick red edges indicate the failing dependency.

The Trace Explorer supports queries on service, resource, operation, status, duration, and any indexed span tag, e.g. service:web status:error duration:>500ms.

Retention filters decide which spans are indexed for 15 days; custom filters can target specific services, resources, or tag values for long-term search.

Trace metrics (trace.http.request.hits, .errors, .duration) power dashboards and monitors and can be grouped by service, resource, version, or custom tags.

Error Tracking groups spans with status:error into issues by error type and stack trace, showing first/last seen and affected version.

Deployment Tracking overlays version-change markers on APM graphs, letting you correlate a latency or error spike with a specific deploy.

Watchdog automatically surfaces anomalies such as elevated error rates or latency on a service without requiring a manually configured monitor.

APM dashboards use trace metrics: hits (request count), errors (errored request count), and duration (latency distribution) as the three base measures.

Latency widgets should use percentile aggregations (p95, p99), not averages, because averages hide tail latency affecting a minority of users.

The Service page combines throughput, error rate, latency percentiles, dependency map, and a resource breakdown table in one view.

Distribution metrics (DDSketch) enable accurate percentile aggregation across hosts and time; APM latency percentiles are computed this way.

Breakdown by resource shows which endpoint dominates latency; breakdown by version shows whether a deploy caused a regression.

Correlate APM with logs and RUM by trace_id: clicking a span opens linked logs and, for browser requests, the corresponding RUM view.

Monitors on APM metrics use the same query syntax as dashboards, e.g. avg:trace.http.request.duration{service:web}.rollup(p95) > 1.

Notebooks and dashboards can embed a Trace Explorer query so a saved search becomes a reusable, shareable visualization.

Start from the symptom: high error rate -> inspect status:error spans and Error Tracking; high latency -> inspect the flame graph critical path.

The flame graph shows span duration as horizontal width; the widest child on the critical path is the dominant contributor to total latency.

Broken traces (missing parent spans) usually mean context propagation failed at a boundary: check header forwarding in proxies, queues, or async workers.

N+1 query patterns appear as many short sibling DB spans under one request span; fix by batching or eager loading.

Errors marked on spans carry error.msg, error.type, and error.stack tags; these are searchable and drive Error Tracking grouping.

Compare a slow trace against a baseline trace of the same resource to spot which child span grew; use the version tag to isolate a bad deploy.

Agent connectivity issues (no traces appearing) are diagnosed with agent status and datadog-agent trace-agent logs; common causes are wrong DD_AGENT_HOST or blocked port 8126.

If trace metrics look correct but individual traces are missing, sampling or retention filters are the cause; raise the sampling rate or add a retention filter.