Prometheus at 4.1M series: ranking a cardinality cleanup
Your platform runs a single Prometheus 2.53 scraping 240 targets every 15s, with remote write to Thanos. Active series grew from 800k to 4.1M in six weeks. The process now OOMs roughly every nine hours, recording rules lag four to six minutes behind wall clock, and dashboard queries hit the two-minute timeout. Three sources dominate the growth: an HTTP request histogram with 18 buckets labeled by tenant_id (about 9,000 active tenants) and by route, where route is taken from the raw request path so it contains values like /orders/8f3c1e2a-.../items; a JVM client library exposing per-endpoint metrics for roughly 400 endpoints; and a debug build that added a user_id label to one counter. Remote write plus long-term storage cost has tripled. Product still wants per-tenant latency visibility for top customers, and all SLO dashboards are built on the HTTP histogram, including its route label. The on-call rotation now pages on Prometheus itself.
1. Rank the three sources by how aggressively you would cut them, with reasoning and rough series estimates. Which removals are safe on their own and which need a substitute?
Read the worked answer
Rank by series contributed multiplied by growth potential, not by how strange a label looks. Head memory in Prometheus is roughly a few kilobytes per active series, so at 4.1M series you are already in the 10GB+ range and OOMing is expected, not surprising.
1. route from the raw path - cut first, highest and unbounded. Every distinct UUID creates a new series, forever. Estimate after simply normalising to templates: 9,000 tenants × ~20 routes × 18 buckets = 3.24M, which is most of the current total. Fix at the source: emit the framework's registered route pattern, not the request path. If the code change is weeks away, use metric_relabel_configs with a regex replacing UUIDs and numeric IDs. This is a correctness fix, not just a cost fix - raw paths also make the SLO dashboard's route label useless.
2. tenant_id on an 18-bucket histogram - cut second, bounded but huge. Even after normalisation you are at 3.24M. Keep full-fidelity tenant latency only for a bounded set (top 200 accounts plus an other bucket) and reduce that metric to 6–8 buckets; that lands near 30–50k series. The substitute for the long tail is traces and exemplars, which answer "why is tenant 4,113 slow" far better than a bucketed p99.
3. user_id - drop unconditionally. Small today, guaranteed to grow, no aggregation or alerting value. metric_relabel_configs drop on the label name, plus a CI check so it cannot come back.
4. JVM per-endpoint - keep. 400 endpoints × ~30 metrics ≈ 12k series. Confirm the endpoint label uses templates rather than raw URIs before declaring it safe.
2. A colleague argues the fix belongs in Grafana queries, not in scrape config. Why is that wrong, and what risks does scrape-time relabeling introduce?
Read the worked answer
Cardinality is created at ingest. Every series is instantiated in the head block, written to the WAL, remote-written to Thanos, and indexed for query. Filtering in Grafana happens after all of that cost has been paid. Query-time filtering reduces what a human sees; it does not reduce RAM, rule evaluation time, remote write volume, or storage. It also does not fix the recording rules, which are what is actually lagging - they evaluate over series that still exist.
Risks and mitigations for metric_relabel_configs:
- You are re-implementing application semantics in a regex. A UUID pattern that misses a new ID format reintroduces the explosion silently. Prefer emitting the route template from code; treat relabeling as an interim measure with an owner and a removal date.
- Relabeling hides the source. Tenants of the platform keep shipping high-cardinality labels because nothing fails. Add a cardinality budget per job with an alert on
prometheus_tsdb_head_seriesbyjob, and a CI check that scrapes a build's/metricsand counts distinct label values. - Dropping a label can break consumers.
user_idmay be on someone's ad-hoc dashboard. Inventory dashboards and recording rules before dropping. - Relabel rules only apply to new samples. Existing series age out per retention; expect a lag of one retention period before the head block actually shrinks.
- Do not relabel away something an alert depends on. Removing a label used in an alerting expression silently makes that alert never fire, which is a worse outcome than the OOM.
3. Product insists on per-tenant p99 latency and will not accept removing the label. What options keep the capability without 3M series, and what are the trade-offs?
Read the worked answer
Option A - narrow the histogram, not the dimension. Six buckets instead of eighteen, on one metric only, restricted to top-N tenants plus other. Cheap, keeps PromQL aggregation, but p99 from broad buckets is an interpolation with real error, and long-tail tenants silently disappear. If you take this, say out loud that p99 for a tenant outside the top N does not exist.
Option B - separate tenant metrics from the SLO metric. Keep tenant_id on low-cost counters (requests, errors) and keep the histogram tenant-free for SLO math. You can compute per-tenant error rates exactly and per-tenant latency approximately. Usually the right answer: most "p99 per tenant" requests are really "is this tenant having a bad time", which errors answer.
Option C - native histograms. Exponential buckets store one series per metric-label set rather than one per bucket, which removes the bucket multiplier entirely. Verify the stability status in your exact Prometheus and Thanos versions before committing, and check that your dashboards and recording rules work with the new sample type.
Option D - exemplars and traces. Attach exemplars to the histogram and answer tenant-specific latency questions by pivoting into a trace. Exact, unbounded, and does not touch the TSDB - but requires tracing to be in place and makes dashboards slower to build.
Option E - summaries. Per-tenant summaries avoid the bucket multiplier, but quantiles are not aggregatable across replicas or pods, so a tenant with three instances gets three meaningless p99s. Only viable if each tenant is served by exactly one instance.
A defensible combination is B plus D: exact per-tenant counters, traces for latency investigation, and a broadened histogram kept tenant-free for the SLO.