Prometheus at 800k Series With a Noisy Pager
A single Prometheus server scrapes 400 targets every 15 seconds and holds about 800,000 active series against a 500 GiB volume with 30-day retention. A platform-owned exporter exposes request_duration_seconds with a user_id label carrying roughly 120,000 distinct values and an http_path label that includes raw IDs. A rule group with 60 recording rules takes about 45 seconds to evaluate; rule_group_iterations_missed_total climbs steadily and dashboards show a gap every 15 minutes. Ingestion is at 85% of what the node can sustain. Alertmanager pages roughly 300 times a night, and about 90% of those pages are PodNotReady from a CronJob that starts and stops by design, plus duplicate PodNotReady alerts from nodes that were already reported down. On-call has started acknowledging pages without reading them.
1. Identify the cardinality drivers and write the relabel configuration you would apply. Explain what data is lost and how you decided it was acceptable.
Read the worked answer
Cardinality is the product of label values, so two labels dominate: user_id (120k values) multiplied by http_path (unbounded, because it embeds IDs) multiplied by the existing histogram buckets and routes. The exporter metric is effectively 120k distinct series for a single conceptual signal.
Apply drop-at-scrape relabeling, which is irreversible and cheapest:
scrape_configs:
- job_name: platform-exporter
metric_relabel_configs:
- source_labels: [__name__]
regex: request_duration_seconds
target_label: __tmp_hist
- source_labels: [__tmp_hist, http_path]
regex: ';.*/[0-9a-f]{8,}.*'
action: drop
- regex: user_id
action: labeldrop
Better than dropping the path series is normalising http_path to the route template at the application, which preserves the signal. user_id is the real decision: dropping it removes the ability to answer any per-customer question forever. Ask whether any SLO, alert, or incident runbook queries that label. If none does, drop it and keep a sum by (le, route) recording rule at 30 seconds evaluated over the aggregated data. If a billing team does query it, that belongs in a separate, sampled pipeline, not in the shared Prometheus.
2. Size the storage and retention change. Show the estimate and state the assumptions explicitly.
Read the worked answer
Samples per series per day = 86,400 / 15 = 5,760. At 800k series that is 4.6 billion samples per day. Compressed Prometheus blocks typically run about 1-2 bytes per sample depending on label churn and histogram content; using 1.5 bytes gives roughly 7 GB/day of block data, or about 210 GB over 30 days, plus index, WAL, and compaction headroom of 20-30%, landing near 270 GB -- which fits 500 GiB, but the volume is not the binding constraint. The binding constraints are head series memory and rule evaluation time.
After dropping user_id and normalising paths, a realistic outcome is 150k-250k series: 1.4 billion samples/day, roughly 2 GB/day, ~65 GB per 30 days. That buys headroom for both memory and the 45-second rule group.
Treat these as estimates and verify with prometheus_tsdb_head_series, prometheus_tsdb_compaction_chunk_size_bytes, and actual on-disk block sizes after a week. Two additional levers beat retention cuts: split rule groups so a slow group does not delay the rest, and evaluate heavy recording rules at 60 seconds rather than 15. Cutting retention to 15 days halves disk use but destroys capacity planning and incident forensics, so it is the last lever, not the first.
3. Restructure the alerting so on-call trusts the pager. What specific mechanisms address the CronJob noise and the duplicate node alerts?
Read the worked answer
The CronJob noise is a signal with a for duration of zero pointed at a workload whose normal state is not-ready. Two fixes, in order of value: do not alert on a workload's ordinary lifecycle -- scope the alert by label so deliberate start/stop jobs are excluded, or route them to a low-severity channel; then add for: 10m so flaps never reach a human. keep_firing_for handles the reverse problem of an alert resolving and re-firing on every scrape.
The duplicate PodNotReady on down nodes is an inhibition problem: define an inhibitor where an active NodeDown (or KubeletDown) suppresses PodNotReady alerts whose node label matches. Alertmanager grouping by alertname, namespace, and cluster collapses the remaining burst into one notification.
Add the dimension most teams miss: alert on the absence of data. If Prometheus stops scraping a target, PodNotReady simply stops firing and the system looks healthy. An absent(up{job="kubelet"})-style alert, and a watchdog alert that always fires and is routed to a dead-man's-switch receiver, catches a broken monitoring pipeline rather than a broken workload. Finally, review page volume weekly and treat any alert nobody acted on as a bug to be fixed or deleted.