Why the p99 Panel Disagrees With the Traces
A service runs on 12 replicas, scraped every 30 s, and exposes a Go histogram http_request_duration_seconds with buckets at 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10 and +Inf. The latency panel query is histogram_quantile(0.99, sum by (le) (http_request_duration_seconds_bucket)) and it reports p99 ≈ 4.8 s. The tracing system, sampling 10%, reports p99 ≈ 700 ms for the same hour. Separately, a throughput panel sum(rate(http_requests_total[5m])) dips by about 35% for two or three minutes after every rolling deployment; deployments use maxUnavailable 25%, and pods take roughly 20 s to start accepting traffic. An engineer proposes replacing rate with irate to "remove the dip".
1. Explain why the p99 panel is wrong and give the correct query. What does the comparison with traces tell you?
Read the worked answer
The query omits rate(). The _bucket series are cumulative counters since process start, so histogram_quantile over raw counters computes a lifetime quantile: it is weighted by every request since the pod booted, including an old bad release and long client-side timeouts, not by the last hour. Traces, which only look at the current hour, see 700 ms. Both are internally consistent; only one answers the question.
Corrected query:
histogram_quantile(0.99,
sum by (le) (rate(http_request_duration_seconds_bucket[5m])))
Two details matter and are easy to conflate:
rate()converts the cumulative counter into a per-second increment profile, which is what the quantile function expects.by (le)preserves bucket identity while summing across the 12 replicas. Summing histograms across instances is valid; it is exactly the operation that is invalid for summaries, whose per-instance quantiles cannot be combined.
When a coarse bucket-derived p99 disagrees with traces, believe traces for tail latency and fix the query before fixing the buckets.
2. After the fix, what accuracy limits remain, and would you change the bucket layout?
Read the worked answer
histogram_quantile assumes observations are spread uniformly within each bucket and interpolates linearly between the boundaries it finds. Two consequences:
- A p99 of ~700 ms falls inside the 0.5–1.0 bucket, where interpolation is reasonable. The error grows with bucket width, so very wide bounds give a confident-looking but imprecise number.
- Anything above the highest finite bound lands in
+Inf. If a meaningful share of requests exceeds 10 s, the function cannot place them and effectively caps the estimate at the second-highest bound - here 10 s.
Check the tail share first:
sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
If +Inf carries more than a fraction of a percent, the reported p99 is an understatement you cannot correct by arithmetic. For a service with a 10 s timeout and a 45 s worst case, add buckets around the real distribution - 0.75, 1.5, 3, 7.5, 15, 30, 45, 60 - and keep +Inf. Changing buckets requires a restart and creates a series discontinuity, so also note that native histograms (exponential buckets) remove manual bucket choice entirely.
3. Explain the 35% throughput dip after each rolling deployment and evaluate the `irate` proposal.
Read the worked answer
The dip is an artefact of scrape resolution plus counter resets, not lost traffic:
- With maxUnavailable 25% and ~20 s to readiness, roughly a quarter of the replicas are not accepting traffic and may refuse scrapes; their series are absent, so the sum loses their share for the window in which they are missing.
- Each restarted pod resets its counter to zero.
rate()handles a reset correctly by assuming the counter restarted at zero, but it cannot reconstruct the requests between the last pre-restart scrape and the restart - up to one 30 s interval per pod, and several pods cycle within the same 5 m window. - The 5 m window keeps the transition visible for five minutes after the last pod returns, which is why it looks like a sustained 35% drop.
irate() would make it worse. It uses only the last two samples, so a reset or a missing scrape between them produces a spike or a zero, and per-series irate values summed across replicas are not a meaningful aggregate rate.
Keep rate(...[5m]). To reassure yourself, corroborate against the load balancer's own request counter and plot up alongside the throughput panel during deploys - the dip should coincide with up == 0, not with a drop in served requests.