Latency spikes disappear on a seven-day panel
A product team has a time series panel showing p95 request latency from Prometheus. The dashboard time range is Last 7 days, refreshes every minute, and the panel has Max data points set to 100 and Min interval set to 30m by a previous owner. The graph looks almost flat around 240 ms. During a 12-minute incident two days ago, latency reached 1.8 s. A colleague opens Explore with the same PromQL for that 20-minute window and sees the spike. The panel query is:
histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{job="checkout"}[5m])))
The rate window is 5 minutes. The team needs the 7-day view to retain sub-15-minute spikes without making the browser slow or overloading Prometheus.
1. Why does the panel miss the 12-minute spike while Explore shows it?
Read the worked answer
The difference is query resolution. Grafana turns the dashboard time range into a Prometheus step, constrained by the panel's Max data points and Min interval. With a 7-day range, Max data points 100, and Min interval 30m, Grafana evaluates the query at intervals that can be longer than 30 minutes. Prometheus returns one value per step, so a 12-minute spike can fall between evaluation points and never appear in the returned series. The 5-minute rate window does not guarantee that every spike is sampled; it only defines the lookback when an evaluation does occur.
Explore showed the spike because the time range was only 20 minutes, which produced a much smaller step. The data is not missing from Prometheus. The dashboard is undersampling it. The flat 240 ms line is plausible because each returned point is a p95 computed at a coarse evaluation time, not an average over the whole gap, but the spike is skipped entirely.
2. What panel and query settings would you change, and what tradeoff does each change introduce?
Read the worked answer
First remove the accidental Min interval of 30m and set it to a small value or leave it unset, then raise Max data points enough for the 7-day range. A step of 1 to 5 minutes over 7 days means roughly 2,000 to 10,000 points per series, which is manageable for one panel but costly if many panels query at once. Use $__rate_interval instead of a hard-coded [5m] so the rate window scales with the step and does not become smaller than the evaluation interval.
There is a direct tradeoff: finer steps catch short spikes but increase Prometheus load and browser rendering time. For a single incident-review panel, 1-minute resolution is reasonable. For an overview dashboard with dozens of panels, keep a coarser step and link to a focused panel or Explore view for investigation. A recording rule can preserve spike-sensitive aggregates if the team needs both low load and high fidelity, but the setup says that is not available this quarter.
3. How would you prove the fix works without waiting for another incident?
Read the worked answer
Use the historical incident window as a test fixture. Set the dashboard to the 7-day range after changing the settings, then inspect the query inspector to confirm the actual step sent to Prometheus is small enough to sample a 12-minute event. Compare the panel's series against Explore over the same 7 days and the narrower 20-minute window. The known spike at 1.8 s should appear in both.
Also test a control period with no incident to ensure the panel is not inventing spikes through interpolation. Check dashboard load time and Prometheus query duration before and after, because the fix is only acceptable if it meets the team's performance budget. Record the step and max data points in the dashboard description so a future editor does not reintroduce the coarse Min interval.