The canary with a flat median
payments is a Java service being rolled out as a 15/85 canary between v2.3.0 and v2.2.14. Median latency is unchanged at 118 ms for both versions and the error rate is flat at 0.4%. But the p99 for v2.3.0 is 2.7 s against 890 ms for v2.2.14, and the difference is concentrated on a single resource, POST /v1/charges. The application tags spans with payment.currency. A slow trace from v2.3.0 opens on a 3.1 s servlet span whose waterfall shows 47 sequential postgres.query children, each about 5 ms, all against the same statement. A comparable trace from v2.2.14 shows three queries. The team's release manager argues that because errors are flat and p50 has not moved, the canary should be promoted to 100% before the weekend. Traffic mix across currencies has shifted in the last two weeks as a new market went live.
1. Design the investigation that gets from 'v2.3.0 is slower' to a defensible cause, using the tags and signals described. State the filters you would apply, what you would read off a single trace, and how you would quantify the effect.
Read the worked answer
A flat p50 with a 3x p99 is a tail phenomenon, so every step must compare distributions, not central tendencies.
Scope and split. Query service:payments env:prod resource_name:"POST /v1/charges", split by version. Look at the latency distribution rather than a single percentile, and compute the fraction of requests above roughly 1 s per version - that gives an effect size in requests, which is what a rollback decision needs.
De-confound traffic mix. Add @payment.currency as a group-by. The new market launch means the two versions may not be serving the same currency mix; if v2.3.0's canary slice is skewed toward the new currency, an overall p99 comparison is confounded. Compare each version within each currency.
Read one trace properly. The 3.1 s span is not database slowness - each query is ~5 ms. It is query count. Forty-seven sequential round trips at 5 ms each is ~235 ms of database time; the rest of the 3.1 s needs explaining too, so check the gap between successive query spans (connection acquisition, application logic between calls) rather than assuming the queries account for the total. Forty-seven identical statements against one statement is the signature of an N+1 loop, and the count is what changed between versions.
Quantify. Create a span-based metric counting postgres.query spans, or use trace analytics to count spans per trace, and ratio it against request hits on the same resource and version. That turns a narrative into a metric you can monitor after the fix. Confirm the loop location with the Continuous Profiler.
2. The release manager wants to promote to 100% because errors are flat. What is wrong with that inference, and what analysis would settle whether this is a v2.3.0 regression or an exposure effect?
Read the worked answer
The inference conflates two SLIs. A flat error rate says failures are not increasing; it says nothing about a tail latency regression that turns into gateway 504s, client-side aborts, and abandoned checkouts. The latency is the user-visible harm here, and the error rate is measured at the service, where a 3.1 s response is still a success. Check the gateway's 504 count and client disconnect rate over the same window before concluding 'no errors, no harm'.
Regression vs exposure. The two hypotheses make different predictions:
- Regression: v2.3.0 is slower for the same currency. Prediction - within each currency bucket, v2.3.0's tail is worse than v2.2.14's.
- Exposure: v2.2.14 is equally slow on the new currency but rarely receives it, because the canary routing or the market launch sends that traffic to v2.3.0. Prediction - v2.2.14's non-USD tail is also ~2.7 s, just on a small sample.
The decisive query is p99 (and share-above-1s) by version × @payment.currency. If v2.3.0 is worse within the same bucket, hold the rollout - a 3x tail regression on the highest-value endpoint is not promotable. If v2.2.14 shows the same tail where it has enough samples, the real event is the market launch, and the 47-query loop is a pre-existing defect newly exposed; still worth fixing, but rollback will not help.
Watch the sample size: at 15% of traffic, per-currency p99 estimates get thin fast. Report counts alongside percentiles, and prefer share-above-threshold over p99 when buckets are small.