Retries Turn a Timeout into Duplicate Charges
Your payments API exposes POST /v1/charges. The VirtualService for payments has a 5s timeout, retries with attempts: 3, perTryTimeout: 2s, and retryOn: 5xx,reset,connect-failure. The downstream card processor intermittently takes 2.5s to 3s and returns 200. During a traffic spike, the payments pods hit their DestinationRule connection pool limit, and some requests return 503 from the sidecar. Clients report duplicate charges, and the processor confirms three charge attempts for a single client order ID. The payments service is not idempotent: it writes a new charge row before calling the processor. Prometheus shows a spike in istio_requests_total with response_code 503 and retries. The team wants to keep retries for safe GET endpoints and stop duplicate charges on POST.
1. Explain how the retry policy produces duplicate charges even though the client sent one request. Which retry conditions and timeouts matter, and why does a 503 trigger another attempt?
Read the worked answer
Envoy retries at the sidecar, not the client, so one client POST can become multiple upstream POSTs. Three mechanisms interact:
perTryTimeout: 2sis shorter than the processor's 2.5-3s response time. Envoy cancels the first attempt at 2 seconds and opens another. The payments server may continue processing and writing the charge row even after Envoy gives up, so the cancellation does not undo the side effect.attempts: 3allows two retries after the initial attempt, producing up to three charge attempts.retryOn: 5xx,reset,connect-failureincludes 5xx responses. When the connection pool limit causes a 503, Envoy treats it as retryable and resends the POST body to another payments pod. Envoy does not know the request is non-idempotent.
The 5s overall timeout also matters: it caps the total time across attempts, but it does not prevent the server from completing a charge after Envoy has already retried. Prometheus shows 503 and retry counters because the retry is visible at the sidecar, while the duplicate charge is visible only in the payments database and processor logs. Fixing only the connection pool would reduce 503s but would not address the perTryTimeout cancellation path.
2. Propose a safe configuration: separate routes for GET and POST, idempotency, timeouts, and circuit breaking. How would you validate before production?
Read the worked answer
Split the route by method and path. For POST /v1/charges, disable retries entirely or restrict them to reset only if the application is idempotent; do not set a perTryTimeout below the processor's p99. Set the route timeout above the worst-case processor latency, for example 6-8 seconds, so Envoy does not cancel a request that is still completing. For GET endpoints, keep attempts and retries as needed.
Application-level idempotency is required regardless of mesh settings. Require an Idempotency-Key header, store it with a unique constraint before writing a charge row, and return the existing result for duplicate keys. This protects against retries from any layer, including client retries and gateway retries.
Add DestinationRule connection pool limits that fail fast instead of queueing indefinitely: set http1MaxPendingRequests, maxRequestsPerConnection, and outlier detection appropriate to the replica count. Do not rely on retries to solve capacity. Use maxEjectionPercent so circuit breaking does not remove all payments pods.
Validate in staging with fault injection: inject 2.5s latency into the processor, force 503s from one payments subset, and run a load test with unique and repeated idempotency keys. Assert one charge row per order ID, not one per HTTP attempt. Before production, query envoy_cluster_upstream_rq_retry and istio_requests_total to confirm retries are zero for POST and unchanged for GET. Roll out to one canary pod first and compare charge-row counts against request counts.