Checkout latency after an HTTP retry change
An e-commerce company runs checkout-api (Node.js) on Kubernetes with payment-service (Java) and an external PSP reached over HTTPS. After release 2026.03.2, checkout-api p95 latency rises from 380 ms to 2.9 s while throughput stays at 410 req/min and error rate stays flat at 0.3%. The release only changed the HTTP client: timeout 2,000 ms, retries 3, exponential backoff. Distributed traces show checkout-api spans around 1.9 s for payment-service, and payment-service spans around 1.8 s in an external call named psp.authorize; the PSP is not instrumented with APM. The service map shows no new services. Pod and node CPU are normal. The PSP status page reports no incident. Product wants to know whether to roll back, raise the timeout, or add capacity. The on-call engineer must decide before the next peak and keep the PSP integration visible for future incidents.
1. Using APM 360 evidence, how would you separate delay inside `checkout-api` from delay in `payment-service` and the PSP? What views or queries would you use, and what conclusion would you draw?
Read the worked answer
Localize before changing anything. In a slow trace waterfall, compare the parent span duration with each child span and its exclusive time. If checkout-api's own span is about 100 ms but the payment-service child is about 1.9 s, checkout CPU and pod capacity are not the constraint. Drill into payment-service traces and isolate psp.authorize; if that external span is about 1.8 s while internal Java work is about 100 ms, the constraint is the PSP call. Use the service map to confirm no new uninstrumented hop appeared, and query Span data around the deployment marker, for example p95 duration grouped by span.name and peer host before and after 2026.03.2. Flat throughput, flat errors, and normal CPU argue against capacity. The new retry configuration is the likely amplifier: a 2 s timeout with three retries can turn one slow PSP response into seconds. I would roll back or cap retries and add a circuit breaker, not add capacity. Evidence that would change my conclusion: high checkout-api exclusive time, CPU throttling, or a new uninstrumented dependency.
2. What alerting or service-level change would you make so this is caught before customers see 2.9 s p95, without paging on every ordinary PSP slowdown?
Read the worked answer
Put the customer outcome first, then the dependency signal. Create a service level for checkout-api with an SLI such as successful checkout requests completing under 1 s. Alert on multi-window error-budget burn rather than a raw p95 page. That catches sustained degradation while tolerating a single short PSP dip. Add a second, lower-severity condition for the external call: an anomaly or static threshold on psp.authorize p95, scoped to payment-service, plus failure rate. Add a synthetic API check against a representative PSP authorization path from two or three regions. Do not page on every PSP slowdown; route dependency alerts to the payment team and use the checkout SLO to decide customer impact. Review after two weeks: if burn-rate alerts fire too late, tighten the SLI latency threshold or add a short-window fast-burn condition. If they fire too often, keep the SLO but downgrade dependency alerts to tickets until PSP latency affects checkout success.
3. The PSP cannot be instrumented. How would you make it visible and prevent a repeat?
Read the worked answer
Treat the PSP as an external dependency with explicit instrumentation and ownership. Wrap the outbound call in a custom segment or span and add attributes such as PSP endpoint, HTTP status, retry count, and timeout flag. That gives you latency and failure visibility even when the PSP has no agent. Keep the PSP on the service map as an external service and add a dashboard showing p95, error rate, timeout count, and retry amplification for psp.authorize. Add a synthetic API check for a safe authorization or health endpoint from the regions that matter, and tie it to a dependency SLO. Create a runbook that says what to do when PSP latency exceeds threshold: disable retries, open circuit, fall back to alternate processor, and notify the PSP contact. Review on-call routing so the payment team owns the dependency alert while the checkout team owns the customer-facing SLO. Validate with a game day: simulate a 2 s PSP delay and confirm the dashboard, alert, and runbook lead to the correct action without adding capacity.