Pushgateway staleness and a histogram cardinality blow-up
A data platform runs 600 cron jobs. Each job pushes completion metrics to one shared Pushgateway with labels job, instance, and schedule. After jobs finish, roughly 180,000 series remain in the Pushgateway. A JobFailed alert based on job:last_success_timestamp fires for jobs that actually completed, and the on-call has muted the alert. Separately, the Go API team added a histogram api_request_duration_seconds with labels tenant_id (50,000 values), user_id (2 million values), route (200 values), and 12 buckets. The target now adds about 1.1 million series per scrape, and Prometheus is OOMing. The team proposes moving all metrics to logs, or keeping only the histogram and dropping the counters. They need a path that preserves tenant-level latency visibility without unbounded series growth.
1. Why does the JobFailed alert fire for successful jobs, and how should the Pushgateway be used correctly?
Read the worked answer
Pushgateway stores the last pushed sample for each grouping key and never expires it. If a job pushes success=1 once and then stops, the metric remains at 1 forever; if the alert checks the wrong timestamp, it can also read old data as current. The Pushgateway is a cache for short-lived jobs, not a general metrics store. Use a unique grouping key per job run or per job identity, and delete the group after the job completes if the metric should not persist. If you need to detect a job that did not run, alert on time() - push_time_seconds{job="..."} > threshold; push_time_seconds is set by the Pushgateway on each push. Set honor_labels: true in the scrape config if the pushed metrics already carry job and instance labels, otherwise Prometheus will rename them. Do not use Pushgateway for long-running services. Cleanup is essential: a lingering failed marker will keep firing after a later success unless the same grouping key is overwritten.
2. How would you reduce the histogram cardinality while keeping tenant-level latency visibility?
Read the worked answer
The series count is roughly the product of label values and buckets: 50,000 tenants times 2 million users times 200 routes times 12 buckets is unbounded, and even a single scrape is enormous. user_id is the first label to remove; it is a request identifier, not an aggregation dimension. Put user_id in logs, traces, or exemplars. For tenant_id, keep it only if the number of active tenants is bounded and the business needs per-tenant SLOs; otherwise aggregate to a tier or top-N tenants and drop the rest. Route can stay if it is bounded; 200 is manageable. Reduce buckets to the few that matter for the SLO, and consider native histograms, which replace bucket series with a single series per label set plus a sparse bucket structure. If tenant-level visibility is mandatory, create a separate metric with only tenant_id, route, and le, and enforce a series budget in CI. Validate with count by (__name__) before and after.
3. Should the team move all metrics to logs or drop the counters, as proposed? What is the better decision?
Read the worked answer
Neither extreme is appropriate. Metrics are for cheap aggregation, alerting, and SLOs; logs are for high-cardinality detail and event inspection. Moving all metrics to logs would remove the low-cost time series that alerting depends on and typically increases cost and query latency. Dropping the counters would remove request rate, error ratio, and saturation signals, leaving only latency. The better decision is to keep the bounded counters (service, route, status_code) and the histogram with reduced labels, while moving user_id and other unbounded identifiers to logs or traces. Add exemplars to the histogram so a latency bucket can link to a trace. Establish a series budget per job and a review step for new labels. If the team lacks tracing, start with structured logs keyed by request_id and use the metrics to find the affected route and tenant, then query logs. This preserves both the SLO and the debugging path.