Lumen Pay: Cardinality Explosion in the Metrics Backbone
Lumen Pay, a payments provider, runs Prometheus with 2.4 million active time series scraped every 15 seconds, up from 300,000 six months ago after a team added a per-customer identifier as a label. Raw retention is 15 days. The anomaly detection model queries multi-day ranges across all series to build features; those queries now take around 40 seconds and frequently time out, so the model silently scores only whatever partition responded first. A vendor advises 'just downsample everything to five minutes', which will cut storage by roughly 90% and make queries fast. Meanwhile the SRE team reports that several sub-two-minute latency spikes during checkout were missed by the model in the last month, and the model's 3-sigma thresholds across all series currently produce a flood of flags that engineers ignore.
1. Diagnose the problem and evaluate the downsampling proposal. What would you do instead, and what does downsampling cost?
Read the worked answer
The growth is a label-cardinality problem, not a storage problem: a customer identifier is unbounded, so series count tracks customer count. At 2.4 million series and a 15-second scrape the pipeline is ingesting roughly 160,000 samples per second, and a naive multi-day range query touches all of it, which is why 40 seconds and partition truncation appear.
Downsampling to five minutes would fix query cost and permanently destroy the signal. Checkout latency spikes shorter than five minutes either vanish or become a mild average, which is precisely the class of incident they already missed. It also relocates the damage: the model would train and serve on smoothed data whose variance is wrong.
Better order of attack: remove the unbounded label and record per-customer aggregates via recording rules with a capped label set; keep raw resolution only for the top tenants and for latency signals; precompute model features as recording rules instead of querying raw ranges at inference; tier retention, raw for three days and one-minute for thirty. Downsampling stays as a last resort for archival only, never for model input.
2. Why is 'downsample in training data only' a trap? Frame the answer in terms of training/serving skew.
Read the worked answer
Because the model would learn one distribution and be scored on another. Trained on five-minute aggregates, it learns the mean and variance of smoothed series; served on 15-second scrapes, the same threshold sees raw jitter and fires constantly, or the score scale shifts so thresholds no longer mean what the evaluation said they meant. Worse, the failure is invisible: predictions still come out, they are just wrong.
There is a second skew with the same shape - aggregate-then-detect versus detect-then-aggregate. Flagging five-minute buckets changes the false positive rate and delays detection past the spike.
The rule is one transformation path shared by training, backtesting, and serving, versioned alongside the model. If the serving path must change for cost reasons, the model is retrained and re-evaluated on the new path before promotion, not patched around.
3. Engineers already ignore the flags. Quantify why, and redesign the alerting approach.
Read the worked answer
Run the arithmetic. A 3-sigma threshold on a roughly normal variable flags about 0.27% of evaluations. Across 2.4 million series, that is around 6,500 flags per evaluation window purely from normal variation - a multiple-comparisons problem, not a sensitivity problem. Add non-normal, heavy-tailed latency distributions and the real number is worse. No human can triage that, so ignoring it is rational behaviour.
Redesign in four moves. First, rank instead of threshold: surface the top-k scores per service per interval, where k is what the on-call can act on. Second, use per-series adaptive baselines conditioned on time of day and weekday rather than one global sigma. Third, apply a false-discovery-rate correction across the group so the family-wise flag count is controlled, and group correlated series so one incident yields one alert. Fourth, validate against confirmed incidents and report precision at k, since a detector that never flags is trivially precise and useless.