Insight Discovery with Datadog APM
Insight discovery with Datadog APM
This domain tests how you find the right trace among millions: Trace Explorer queries, facets, measures, Live Search versus indexed spans, retention filters, saved views, and service-level discovery through the Service Catalog and Service Map.
Search mechanics you must know
The Trace Explorer query bar accepts key:value terms and @tag:value for span tags. Common queries include service:checkout, env:prod, resource_name:GET /orders, status:error, and @http.status_code:500. Live Search returns all spans from roughly the last 15 minutes, which makes it the fastest way to confirm a fresh issue. Older data requires indexed spans, which are governed by retention filters. A retention filter is a query plus a sampling rate that decides which spans are indexed and kept for the retention period; it is not the same as ingestion sampling. Default retention filters typically capture errors and rare or high-latency spans, but custom filters are needed for business-specific traffic.
Facets and measures
A facet is a tag or attribute you have explicitly made searchable and groupable, such as @error.type or @http.method. A measure is a numeric field you can aggregate, such as duration. Custom span tags are not automatically facets; you must create the facet before you can group by them. Saved views preserve a query, time range, and visual settings for recurring investigations.
Worked example
Checkout errors spike. In Live Search, run service:checkout env:prod status:error. Group results by @error.type and by resource_name. One endpoint, POST /checkout, dominates. Open a representative trace and follow the flame graph into the downstream payment service, where a db span shows a timeout. Add a saved view for this query and, if the pattern must be searchable tomorrow, create a retention filter scoped to service:checkout env:prod status:error rather than assuming Live Search retains it.
Traps
Expecting a custom tag to be groupable without creating a facet. Confusing retention filter sampling with head-based ingestion sampling. Searching for yesterday's trace in Live Search. Using status:error when the failure is only visible as @http.status_code:503. Creating facets on high-cardinality fields such as request IDs, which fragments results and slows queries.