Alert Storm and Impossible rate() After a Clock Step

Prometheus timestamps samples with the server's clock, and the TSDB requires per-series ingestion in increasing time order.

• While the VM clock was 12 minutes behind, incoming samples arrived with timestamps older than the last stored sample for that series. The head block rejects them as out-of-order, so the samples are dropped and panels show gaps.
• With no new samples, the last good sample ages past the lookback delta (5 m by default). The series goes stale, up evaluates to 0, and InstanceDown fires - roughly 220 times, one per target whose series went stale. The targets were healthy the whole time.
• After the step forward, the same series jump 12 minutes into the future. That is legal and ingestion resumes, but the stored timeline now contains a discontinuity that every range function will read as real.
• The duplicate delivery is unrelated configuration: Alertmanager fans out to every matching receiver. It deduplicates repeated notifications for the same alert within one receiver path, but two receivers each deliver. Use a single receiver with fan-out, or inhibit_rules that suppress the secondary route while the primary fires.

Both functions assume the counter is continuous inside the range window.

• The spike: Prometheus takes the first and last samples in the range, divides the delta by the elapsed time, then extrapolates to the window boundaries. With a 12-minute discontinuity the 'last' sample sits far ahead of where the counter should be for that wall-clock instant, so the extrapolation overshoots - the observed ~3× is the extrapolation factor, not extra traffic.
• The apparent resets: when a sample is lower than its predecessor, rate() classifies it as a counter reset and assumes the counter restarted from zero. A backwards timestamp jump produces exactly that pattern, so increase() yields nonsense, including negative-looking values from reset compensation.

Handling it:

1. Do not trust, backfill, or average across the step. The data around it is unrecoverable.
2. Remove it so recording rules and dashboards stop propagating it: use the admin delete_series API with the affected matchers and a start/end covering the step, or accept the loss and pin dashboards to a range that starts after the correction.
3. Re-run any recording rules that consumed the bad window, and annotate the dashboards with the incident so the spike is not read as a capacity event.
4. Keep for: clauses on rate-based alerts - a few minutes of bad extrapolation should not page anyone.

Treat the monitoring server as a production dependency with its own failure modes.

Clock

• chrony with makestep 1.0 3: the first corrections after boot or restore are stepped quickly, later ones are slewed so Prometheus never sees a jump of minutes.
• Order the service after time-sync.target (Wants= and After=), so scraping does not start against an unset clock.
• Alert on abs(node_timex_offset_seconds) > 0.5 for 5 minutes.

Alert rules

• Add for: 5m (at least 2× scrape interval plus margin) to InstanceDown; a 12-minute step should page once when the problem is real, not 220 times.
• Require a second signal before paging, for example up == 0 combined with the target having been up shortly before, or a node-level heartbeat from a second source.
• Use inhibit_rules so a node-down alert suppresses its derived application alerts, and configure one receiver with fan-out rather than two receivers for the same severity.

The watcher

• An external dead-man's switch - a second host or an off-site check hitting a synthetic always-firing alert every minute - catches a Prometheus that is up but not ingesting.
• --scrape.timestamp-tolerance rejects absurd timestamps at ingestion; out-of-order counters in the server log are an early warning worth alerting on.
• Test the restore path: a restored VM should run chrony to sync before Prometheus starts, and that ordering should be in the unit file, not in a runbook step someone forgets.