Datadog APM & Distributed Tracing Fundamentals: Fifty-two thousand resource names

Fifty-two thousand resource names

storefront-api is an Express application running on 40 Kubernetes pods. It was instrumented with dd-trace-js at startup and no per-route configuration. Spans are named express.request and the resource is the raw URL, so requests to /api/products/8812/reviews and /api/products/4417/reviews produce different resources. Over the past seven days the service generated 52,000 distinct resource names. Two consequences have surfaced: the APM resource list is unusable for grouping or alerting, and a span-based metric the team created on resource_name is nearing its cardinality ceiling. Separately, liveness and readiness probes hit /healthz every five seconds on every pod, which works out to about 1.4 million spans per day - roughly a third of the service's span volume and most of its hits. Apdex for storefront-api reads 0.94 and looks healthy, and on-call wants probe traffic excluded without losing the ability to notice a genuinely failing pod.

1. Rank the available fixes for the resource-cardinality problem and justify the ranking. Distinguish clearly between mechanisms that change metric dimensions and mechanisms that only change what is searchable.

Read the worked answer

The dimension that matters is resource, and it is baked into trace metrics at the point they are generated. Any fix must therefore act before metrics are computed - that is, in the tracer or at the Agent - or the metrics stay contaminated.

Fix it in code (best). Set the resource to the route pattern, not the URL: GET /api/products/:id/reviews. With dd-trace-js you can do this via the Express plugin hook, setting span.resource from the matched route. This collapses tens of thousands of resources into a few hundred, fixes metrics and search simultaneously, and survives new URL shapes. Caveat: unmatched routes (404s, scanner traffic) may not have a route pattern and will still emit raw paths, so add a fallback that buckets unknown paths into a single resource such as GET /api/*.
Agent-side exclusion or normalization. apm_config.ignore_resources drops matching traces at intake, before trace metrics are computed. Good for known-bad endpoints, but it is a regex you now own and must roll out to every pod's Agent.
Retention filters (does not help here). Retention filters decide which traces are stored for search; metrics are still computed from everything the Agent receives. They will not reduce cardinality or clean up apdex. Including them in the fix list is the classic mistake this question is testing.
Ingest-time remapping rules are a legitimate stopgap for known patterns without a redeploy, but they are a maintained translation layer, not a cure, and the tracer keeps emitting unique values.

2. Decide how /healthz traffic should be excluded, and explain the effect of each option on metrics, retention, and the ability to detect a failing pod.

Read the worked answer

The probes are 1.4M spans per day out of about 4.1M - a third of the volume, and because they are fast and never error, they inflate hits and flatter apdex. Apdex in particular is not something you can filter on a dashboard; it is computed per service, so probes must be removed before the metric exists.

Option A - Agent ignore_resources for the probe resources. Drops those traces before they reach the backend, so hits, apdex, and search are all clean. Cheapest immediate fix. Risks: the regex must match the actual resource names (so do the route-pattern fix first, or /healthz/123 variants will slip through), it must be applied to all 40 Agents via config management, and you lose probe latency visibility.

Option B - stop tracing them in code. Cleaner ownership, but needs a deploy and a way to identify probe requests that is not spoofable by clients hitting the same path.

Option C - keep them and filter in every dashboard query. Worst of the three: every widget and monitor must carry the exclusion forever, apdex remains corrupted, and one forgotten query reintroduces the noise.

Detecting a failing pod does not depend on probe traces. Use Kubernetes readiness state and container restart metrics, plus synthetic or uptime checks that exercise a deeper path than /healthz. Liveness and readiness probes are deliberately shallow; they were never a good availability signal, and treating their traces as one is part of why the volume grew unnoticed.