Batch Jobs, Slow Exporters and Misranged Buckets

The Pushgateway is a cache with no expiry. A job that silently stops running keeps reporting its last values forever, so "the job stopped" is invisible - the worst possible failure mode for a reconciliation job.

Three specific consequences here:

• up is always 1, because Prometheus scrapes the Pushgateway, not the job. You have no availability signal for the job at all.
• With honor_labels unset, the pushed job and instance labels are overwritten by the Pushgateway target's own labels, so the four containers collide into one ambiguous series.
• Nothing distinguishes "pushed 30 s ago" from "pushed nine days ago".

Better options, in order:

1. Write a .prom file and use node_exporter's textfile collector, then alert on file age - the metric lifecycle is visible.
2. If Pushgateway is kept: always export a last-success timestamp (reconciliation_last_success_timestamp_seconds) and alert on time() - ... > 1.5 × period, push only once at the end of the run, delete the group at start, and set honor_labels: true.

A 90 s job can also be scraped directly if the scrape interval is shorter than the runtime; the Pushgateway is justified by the 6-hour gap, not by the 90 s runtime.

Every scrape is cancelled after 10 s. Prometheus records up == 0 and stores no samples, while the exporter keeps working on work whose client has already gone. With concurrent scrapes arriving every 30 s, abandoned 40 s requests pile up and starve the process - the exporter becomes the outage.

The handler must become O(1) and never do I/O:

1. Run the billing API call and the 200,000-row aggregation in a background goroutine on a ticker (for example every 120 s), with its own timeout, jitter and backoff.
2. Store the result in an atomic or mutex-guarded struct.
3. Let /metrics only serialize the cached values.
4. Add a freshness gauge, billing_last_success_timestamp_seconds, and alert on it, so a stale cache is loud rather than silently frozen.
5. Honour r.Context() so an aborted scrape actually stops work.

Be explicit about the trade-off: the dashboard now shows data up to two minutes old, which is correct for a slow upstream and far better than showing nothing. Set the scrape interval and the background refresh independently - they answer different questions.

With default buckets topping out at 10 s, every 45 s observation falls into +Inf. histogram_quantile cannot place observations above the highest finite bound, so it returns roughly the second-highest boundary - 10 s - as if that were the p99. The number looks plausible and is a lie in the safest possible direction.

Fix the buckets to bracket the actual distribution, e.g. 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 20, 30, 45, 60, 90, and always keep +Inf. Buckets only need to be dense where the quantile of interest lives.

Keep the histogram, delete the summary, because:

• Histograms are aggregatable: sum by (le) across replicas is valid, so a fleet-wide p99 exists.
• Summaries compute quantiles client-side per pod; averaging or summing them is statistically meaningless, and there is no way to combine 12 replicas' 0.99 values into a correct fleet p99.
• Summary quantiles are fixed at instrumentation time and cannot be recomputed for a different window or a different quantile.

For exact tail truth, use traces - but the histogram with sane buckets is what belongs in Prometheus.