Histogram and error-ratio queries that look plausible
The checkout service runs on 40 pods and exports http_requests_total{service,pod,status_code} and http_request_duration_seconds_bucket{service,pod,le}. Scrapes happen every 30s. A Grafana panel uses histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{service="checkout"}[5m])) and displays 180 ms, while application logs show a p95 of 1.6 s. A second panel uses rate(http_requests_total{service="checkout",status_code=~"5.."}[5m]) / rate(http_requests_total{service="checkout"}[5m]) and shows values around 1.2, which is impossible for a ratio. One pod restarted three minutes ago, resetting its counters. The team also wants an alert when the checkout error ratio exceeds 5% for 10 minutes, but the current expression fires during traffic lulls when the denominator is near zero. They have 5 minutes to correct the queries before an incident review.
1. Write the correct PromQL for p95 latency across all checkout pods and explain the error in the original.
Read the worked answer
Correct: histograms are cumulative, so quantile calculation needs one bucket set per le value. The original rate(http_request_duration_seconds_bucket{service="checkout"}[5m]) returns one series per pod and le; histogram_quantile then groups by all labels except le, producing a separate p95 per pod. Grafana may display one of those lines or aggregate them incorrectly, so the 180 ms value is not the service p95. Use histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{service="checkout"}[5m]))). If you need per-route or per-pod quantiles, keep those labels in both the sum by and the grouping, but do not mix levels. The 5m window at a 30s scrape gives 10 samples, which is acceptable but sensitive to resets; a 10m window gives more stability at the cost of slower detection. Also ensure the histogram buckets cover 1.6s; if the largest finite bucket is 1s, the quantile is capped and cannot report the true tail.
2. Rewrite the error-ratio query and explain why the original can show 1.2.
Read the worked answer
The original divides each 5xx series by each corresponding total series before any aggregation. Because the denominator includes all status codes, the per-series ratio can look small, but when Grafana sums or stacks multiple pods the result can exceed 1; a ratio must be computed after aggregating numerator and denominator. Correct: sum(rate(http_requests_total{service="checkout",status_code=~"5.."}[5m])) / sum(rate(http_requests_total{service="checkout"}[5m])). Guard traffic lulls with a minimum request rate or filter the denominator; if the denominator is zero, the result is NaN, and a comparison in an alert is false, which is usually safer than paging on no traffic, but pair it with a traffic-absence alert. Label both sides consistently; do not filter status on the denominator unless the SLO excludes specific codes. For the 5% for 10m alert, use the aggregated ratio and keep the duration in the alert rule rather than embedding a long window in PromQL.
3. How should the 3-minute-old pod and counter reset affect these queries?
Read the worked answer
rate is designed for counters and detects a reset by treating a decrease as a reset; it extrapolates the increase over the window. A pod that restarted three minutes ago has only about six samples in a 5m window at 30s scrape, so its rate is based on a short and noisy interval. That is not automatically wrong, but it can distort a service-level ratio if the pod carries a large share of traffic. Aggregating with sum before the ratio or quantile reduces the influence of a single pod, because the numerator and denominator include all pods. If a pod disappears entirely, its series become stale after the lookback delta; sum simply ignores it. Do not use delta for counters; it does not handle resets. For alerting, require a minimum request rate or use a longer window so a single restart cannot trigger a page.