Scrape Timeouts Versus Real Target Failures
One Prometheus 2.x server scrapes 1,200 targets every 15 s with scrape_timeout: 10s and evaluates recording and alerting rules every 15 s. The API gateway exposes /metrics from the same process and thread pool that serves user traffic; at peak, rendering that endpoint takes 9–12 s. Two rules matter: TargetDown = up == 0 for 2m, and HighErrorRate = sum(rate(http_requests_total{status=~"5.."}[1m])) / sum(rate(http_requests_total[1m])) > 0.05 for 5m. Operators report three symptoms: TargetDown fires and resolves repeatedly for gateways that are actually serving fine; error-rate panels show gaps and jumpy values; and page-worthy alerts arrive 3–4 minutes after the condition is visible in the logs. Grafana queries use a 15 s step with the same [1m] rate window.
1. Diagnose the flapping `TargetDown` alerts. How do you distinguish "slow" from "down", and what do you change first?
Read the worked answer
up == 0 means the scrape failed, not that the target is down. A 12 s render against a 10 s timeout is a failed scrape on a perfectly healthy process; the next fast scrape sets up back to 1, so the alert flaps in step with load.
Discriminate with the scrape meta-metrics:
- Slow:
scrape_duration_secondsclimbing toward 9–10 s and being cancelled. - Down: connection refused almost immediately, so
scrape_duration_secondsstays near zero whileupis 0.
A useful true-down expression is up == 0 and scrape_duration_seconds < 1.
Fixes, in order of value:
- Serve
/metricsfrom a dedicated listener and goroutine pool so request handling cannot starve it, and precompute or cache expensive collectors - the Go client serializes a handler per scrape, so a slow collector stalls everything. - Keep the timeout below the interval; 10 s against 15 s is already near the ceiling, so raising it is not a real fix.
- Lengthen
for:so a single failed scrape cannot page, and alert on scrape duration separate from availability.
2. Why do `[1m]` rates produce gaps and jumpy values at a 15 s scrape interval, and what window would you use instead?
Read the worked answer
A 1-minute range over a 15 s interval contains at most 4 samples, and rate() needs at least 2. One missed scrape leaves 3 samples; two missed scrapes leave fewer than 2 and the series disappears from that evaluation entirely - that is the gap you see, not a real traffic drop.
With only 2–3 samples the extrapolation to the window boundaries is a large correction factor, so one late scrape or one long render skews the computed slope - that is the jumpiness.
Use a window of at least 4× the scrape interval:
[2m]is the practical floor at a 15 s interval.[5m]for dashboards and for anything that feeds an alert; it survives a missed scrape, smooths partial restarts, and makes thefor:duration meaningful.
Also keep the Grafana step at or above the scrape interval, and prefer $__rate_interval so the window scales with the panel's step. irate() is not the fix - it looks at the last two samples only and is more sensitive to the same jitter.
3. Account for the 3–4 minute delay before the alert reaches the on-call engineer. Which stage would you shorten, and which would you leave alone?
Read the worked answer
The delay is a sum of stages, not a single bug:
- The expression needs a full
[1m]rate window, so detection lags onset by up to 1 minute. - Rules evaluate every 15 s, adding up to one evaluation interval.
for: 5mrequires the condition to hold across that whole span, so firing cannot happen earlier than ~5 minutes after onset.- Alertmanager applies
group_wait(30 s default) before the first notification of a group, andgroup_interval(5 m default) before additions to an existing group are sent.
Total ≈ 1 m + 5 m + up to 5 m.
Shorten the for: - it is the largest single term - but do not do it alone: widen the rate window to [5m] at the same time so the signal is at least as stable as before and the shorter for: does not reintroduce flapping. Reduce group_interval to 1–2 m if new hosts must surface quickly inside an ongoing incident.
Leave repeat_interval long. Do not shrink the rate window to [1m] to buy a minute: that trades latency for gaps and jitter, which is worse than being slow by 60 seconds.