Cardinality Budget for a Payments API
A payments team ships a Go service exposing a single counter, http_requests_total, labelled with method, path, status, customer_id and region. The environment currently has 4 methods, about 60 distinct raw URL paths, 12 status codes, 8,000 active customers and 6 regions. A week after rollout the Prometheus pod (2 vCPU, 8 GiB, 15-day retention) OOM-kills nightly, prometheus_tsdb_head_series reports roughly 1.4e8, and Grafana panels time out. The team's proposal is to raise the memory limit and cut retention to 3 days. Support engineers still want to answer "which customers saw elevated 5xx rates in the last hour?", and the SLO is defined as the share of non-5xx requests per region. The tracing pipeline already samples 10% of requests and carries customer_id as a span attribute; nobody has connected it to the metrics yet.
1. Explain why the server runs out of memory and state the label set you would keep. Show the arithmetic and the resulting series count.
Read the worked answer
Cardinality is the product of the distinct values of every label, because each unique combination is its own time series.
4 methods × 60 paths × 12 statuses × 8,000 customers × 6 regions ≈ 138 million series
Each active series costs head-block memory (index entries plus chunk data) and a postings entry, so an 8 GiB pod dies long before ingestion is even the bottleneck. Every query selector must then walk enormous postings lists, which is why panels time out.
Retention is not the problem: retention governs the on-disk blocks, while the head block holds the most recent ~2 hours in memory. Shortening it changes nothing.
Keep method, status, region, and replace the raw path with the router's template (/v1/payments/{id}), about 20 distinct routes:
4 × 20 × 12 × 6 = 5,760 series
That is roughly a 24,000× reduction. Estimate the product before shipping any new label, enforce the cap at the SDK (OTel views or a metrics wrapper), and keep a metric_relabel_configs labeldrop as a backstop.
2. Where should per-customer investigation live, and does removing `customer_id` weaken the SLO? Justify your answer.
Read the worked answer
The SLO is per region, share of non-5xx - fully computable from method, status and region. Dropping customer_id costs the SLO nothing; it only removes a dimension the SLO never used.
Customer identity is unbounded by nature: the number of customers grows without limit, so any per-customer label is a cardinality time bomb regardless of today's value.
Per-customer questions belong where cost per request is bounded by sampling rather than by distinct values:
- Traces. The pipeline already samples 10% and carries
customer_id, so a support engineer can filter by customer and see the actual slow or failing spans. - Logs. Structured logs indexed on
customer_idanswer error-rate questions for a single customer at query time. - Exemplars. Attach a trace ID exemplar to the request-duration histogram and to a sampled error counter so a Grafana panel can jump from an anomalous bucket straight into the relevant traces.
If a metric dimension is genuinely required, make it bounded and categorical - customer tier, or a static allow-list of the top 20 accounts - never a raw ID.
3. A colleague proposes keeping `customer_id` but only on 5xx responses, via a second counter, arguing "only a few customers error". Evaluate that proposal.
Read the worked answer
It reduces the average but does not bound the worst case, and the worst case is exactly when you need the data.
- During a broad outage nearly every customer errors. The series count then approaches the full 8,000 customers - the same explosion, timed to coincide with the incident when Prometheus is already under load from alerting rules and dashboards.
- It also splits numerator and denominator into two different metric families, so
rate(errors)/rate(total)can no longer be computed from one selector at one consistent label set, and any per-customer ratio becomes a fragile join.
Cheap alternatives that keep the capability:
- A bounded label - tier, plan, or a revenue-ranked top-20 allow-list maintained as config.
- Exemplars on an existing aggregate error counter: sampled, bounded per bucket, and they carry the trace ID back to the customer.
- Compute per-customer error rate from logs or the trace backend at query time.
If the team insists, cap it mechanically with relabeling and add an alert on prometheus_tsdb_head_series growth rate so the next bad label is caught in a day, not a week.