Three widgets that disagree
An SRE has inherited a 'Payments SLO' dashboard and must make it trustworthy before a quarterly review. Three widgets are in dispute. Widget A, 'Requests', uses sum:trace.http.request.hits{service:payments,env:prod}.as_count() and reports 9.1M per day, while the APM service page for payments shows 12.4M for the same day. Widget B, 'Latency', shows p99:trace.http.request.duration{service:payments,env:prod} grouped by service across six services in the request path, and the reviewer wants a single headline p99 for the path. Widget C, 'Error rate', is defined as sum:trace.http.request.errors{service:payments,env:prod}.as_count() divided by the hits figure from Widget A, and reads 1.9%, although the service page reports 0.6%. The payments service serves traffic through both HTTP and gRPC, and its traces include client spans for calls to an external card processor. No metric definition has changed in the last quarter, and nobody can explain why the numbers disagree.
1. Explain the 9.1M versus 12.4M gap and prescribe a query pattern for request count that will not drift out of sync with the service page.
Read the worked answer
Trace metrics are generated per operation name - trace.<span_name>.hits, .errors, .duration, .apdex. Widget A pins one operation name, http.request, so it counts only the spans produced by the HTTP entry point. The payments service also serves gRPC, which emits its own operation name, and possibly a consumer operation. The service page's 'Requests' figure is built from all of the service's entry-point spans, so it is larger by the gRPC share. The ratio 9.1/12.4 ≈ 73% is the HTTP fraction, which is a plausible split for a service with an internal gRPC API.
Nothing is wrong with .as_count() here; a rollup mistake would produce a discrepancy of a very different magnitude and shape, not a stable 27%.
To stop the drift:
- Build the count from the full set of entry-point operations explicitly, e.g. HTTP hits plus gRPC hits, and comment the widget with the list. If your trace metrics carry a span-kind tag, filter to server spans instead of enumerating operation names.
- Reconcile once against the service page and record the expected ratio, so a future divergence is visible as a change rather than a mystery.
- Prefer one canonical definition used by dashboard, monitor, and SLO, and put it in a saved view or template variable rather than retyping it per widget.
Also confirm the time frame and timezone of both comparisons before assuming a semantic gap - daily totals across different rollups are a common false positive.
2. Fix Widget B and Widget C. Explain the statistical error in the latency widget and the metric-definition error in the error-rate widget.
Read the worked answer
Widget B - averaging across services. p99:...{...} by {service} returns six independent time series; there is no 'headline p99' computed by looking at them side by side, and under no circumstances should they be averaged. Percentiles are not additive or averageable: the mean of six p99s is not the p99 of anything, it discards each service's request volume, and it can sit below the true path tail. Options, in order of defensibility:
- Query the ungrouped percentile -
p99:trace.http.request.duration{service:payments,env:prod}without the group-by - to get the p99 over the pooled span population, and state that it is volume-weighted by span count. - Show per-service p99 and a max, labelled as 'worst hop', not as a path percentile.
- If you need a true request-weighted path p99, compute it from a distribution metric or from the edge service's entry spans only.
Separately, mixing client and server spans in one duration metric double-counts each request at every hop. Filter to server spans, or restrict to the edge service's entry operation.
Widget C - mismatched numerator and denominator. The numerator counts every error span in payments, including client spans to the card processor (a downstream 500 becomes an error span inside the payments trace) and including gRPC entry spans; the denominator counts only HTTP entry hits. Different populations, so the ratio is not an error rate. One request with three failing client spans contributes three to the numerator and one to the denominator, so the ratio can even exceed 100%.
Fix: same operation name and same span kind on both sides - e.g. server-side hits and server-side error spans for the HTTP entry point, optionally restricted to 5xx status codes. Track processor failures as a separate SLI; they are a dependency-health signal, not payments' own error rate.