A p90 eight times the p50, and two changes shipped the same week

Client-side elapsed time for the inventory call is roughly the p90 figure minus checkout-service's own local work - call it 2.6 s. The callee's handler takes 180 ms. So on the order of 2.4 s is spent outside the callee's handler, somewhere in the path: connection acquisition from a pool, DNS resolution, TCP/TLS handshake, network round-trip and retransmits, serialization, queueing ahead of the handler, or a retry loop that hides itself as one slow call.

That narrows to two families:

• Time before the request is sent - waiting for a pooled connection, a blocking name lookup, or a connect that has to be re-established because a keep-alive connection was dropped.
• Time between send and handler, or handler and receipt - RTT, packet loss and retransmits, a middlebox or proxy in the path, particularly if it is AZ-specific.

Plus a third that masquerades as both: retries. A 2.4 s gap that clusters tightly around a fixed value is a timeout-plus-backoff signature, not a degraded-but-working path.

Discriminating evidence: does the gap sit before the request leaves the caller (pool wait, DNS) or on the wire (RTT/retransmit); is it uniform across pods or confined to some; is the callee's own handler time unchanged, which it is - that alone rules out inventory-service being slow and points at the caller-side path.

p50 of 320 ms with a p90 of 2.6 s is not a service slowly degrading - it is two populations: most requests are fine, a subset is badly broken. Averages and single global percentiles will keep you chasing a phantom.

Prove it by slicing the same metric rather than reading the aggregate:

• split response time by availability zone, then by process instance/pod, then by caller;
• split by request attribute (endpoint, payload size, tenant, auth type) to see if the slow population is a behaviourally distinct set of requests;
• plot the distribution, not just the percentile - a slow population clustered around a round number (2 s, 3 s) is a timeout, a slow population smeared upward with load is contention.

Why it matters: the fix is almost never "make the service faster" but "find the subset". If the slow population is one AZ, it is network or infrastructure. If it is one pod, it is that pod's pool or resolver state. If it is one endpoint, it is a code path. The shape of the distribution tells you which dimension to slice next - a tight cluster means timeout/retry, a load-correlated smear means resource exhaustion.

Pool exhaustion is the better first bet, but neither is confirmed by the evidence you have.

Pool exhaustion predicts:

• the gap sits on the caller side as time waiting to acquire a connection, before anything is sent;
• the callee sees nothing at all - consistent with the flat 180 ms handler time;
• the slow population scales with concurrency and shrinks when traffic drops, and hits only the pods whose pools are actually saturated, so it is pod-shaped, not AZ-shaped;
• it is often quantized by the acquisition timeout, which fits a bimodal distribution.

A DNS problem predicts:

• spikes tied to connection churn or resolver TTL, so periodic rather than concurrency-correlated;
• often AZ-wide or deployment-wide, because every pod picks up the same resolver config at the same rollout.

Cheap confirmation before any rollback:

1. Correlate the slow population with per-pod concurrency - if it tracks concurrency, it is the pool.
2. Check whether the 2.4 s gap appears before the request is issued (pool wait / name resolution) rather than on the wire.
3. Canary one pod back to the previous pool size and one to the previous resolver config, then compare their p90 and distribution against untouched pods for an hour.

Do not roll back both changes at once - you will learn nothing and will likely re-ship the culprit next sprint.