I85PREP
⌘ K
Datadog Fundamentals study materials
Datadog Fundamentals study resources

Tag Drift and a 3.1M-Series Custom Metric Bill

Datadog Fundamentals · Put your knowledge to work in context.

All case studies

Tag Drift and a 3.1M-Series Custom Metric Bill

Northwind Retail runs 240 hosts - 150 EC2 instances and 90 on-premise VMs - with Agent 7 deployed and configured through Ansible. Over three months the custom metric series count climbed from 900k to 3.1M and the Datadog bill followed. Investigation shows engineers adding host tags by hand in the UI: env:prod, Env:Prod and environment:production all describe the same fleet, so the ops dashboard built on env:prod silently hides 60% of hosts during incidents. A 40-node Kubernetes cluster adds churn: pods restart hundreds of times per day and host counts jump. Application teams cannot change code, and Finance requires per-team cost attribution derived from tags.

1. Fix the tag inconsistency so dashboards and cost allocation are stable. What do you change, in what order, and what breaks during the transition?

Read the worked answer

Diagnosis. Tags typed into the host UI are per-host, unreproducible and unauditable. With 240 hosts and Ansible-managed configuration, UI tags guarantee drift. The three spellings are three distinct tag values, which means three separate sets of time series and one dashboard that filters to a subset of hosts without erroring.

Order of operations matters. Adding the canonical tag before removing the old ones doubles the series count for the transition window, so:

  1. Publish a tag schema - env, service, version, team, region - with a fixed, lowercase value list.
  2. Roll the canonical tags out from Ansible using DD_ENV / DD_SERVICE / DD_VERSION plus DD_TAGS. This is config-only, which satisfies the 'no code changes' constraint and makes tags reviewable in git.
  3. Update dashboard and monitor queries to the canonical values; verify the host map shows all 240 hosts under one env value before proceeding.
  4. Remove the hand-typed UI tags and reject them in review.

During the transition expect dashboards showing partial data and inflated series counts. Announce the cutover window so on-call doesn't chase phantom gaps, and keep the old value alive only as long as the migration takes - every extra day is billed cardinality.

2. Bring custom metric series from 3.1M to under 1.5M without breaking alerting. What do you inspect first, what do you change, and how do you verify?

Read the worked answer

Find the offenders before touching anything. Open Metrics Summary, sort custom metrics by series count, and inspect the tag keys each one carries. In practice most of 3.1M comes from a handful of metrics carrying unbounded tags - request IDs, user IDs, raw URL paths, timestamps, exception messages.

Two levers:

  • Metrics without Limits: for the offending metrics, keep only the tags that dashboards and monitors actually query (env, service, team, status_code) and drop the rest. Series collapse to the dimensions you use. The consequence: anything that queried a removed tag no longer breaks down by it. Grep every monitor and dashboard definition for those tag keys before you apply the change.
  • Agent-side tag allowlisting for DogStatsD-heavy services, so bad tags never leave the host for named metrics.

Verification. Re-check Metrics Summary after a day and run each production monitor's query against the new cardinality. Do not simply delete metrics: you lose history and break alerting you didn't know existed, and the 3.1M is usually a symptom of one team's instrumentation, not 240 hosts' worth of genuine need.