Checkout Latency Spike Under Retry Amplification
An online checkout service runs as three identical Kubernetes pods behind a load balancer. Each pod uses a HikariCP pool with maximum 10 connections; the PostgreSQL primary is configured for 100 connections. Normal traffic is 120 requests per second, and the average database query takes 150 ms. During a sale, p99 latency rises from 180 ms to 2.4 s, error rate reaches 8%, and the database reports 30 of 30 service connections active with 45 threads waiting. Application logs show that each failed request is retried twice by the client and once by the service mesh, and the query timeout is 30 s. Database CPU is 45%, disk latency is 4 ms, and there are no slow-query-plan regressions. The team proposes doubling the pool to 60 connections and raising the database max_connections to 300.
1. Use Little's law to estimate database concurrency at normal load and during the sale including retries. Is doubling the pool the right fix?
Read the worked answer
Normal concurrency is 120 requests/s x 0.15 s = 18 database queries in flight across the service. Per pod, 40 requests/s x 0.15 s = 6 connections, so a 10-connection pool is adequate normally. During the sale, a failed request can consume four attempts: the original plus two client retries plus one mesh retry. Attempted rate is approximately 120 + 120 x 0.08 x 3 = 148.8 requests/s, giving 148.8 x 0.15 = 22.3 concurrent queries. That is above the 18-connection normal figure but below the 30 total connections, which shows queueing and timeout holding, not just raw concurrency. With 30 of 30 active and 45 waiting, the pool is saturated. Doubling each pod to 60 would allow up to 180 connections against a 100-connection database, causing connection thrash and memory pressure. The database CPU at 45% confirms it is not CPU-bound. The fix is to stop retry amplification and bound connection hold time, not to enlarge the pool blindly.
2. Propose concrete pool, timeout and retry settings, and a validation plan that would prove the fix.
Read the worked answer
Set each pod to a 12-connection pool, giving 36 total and leaving headroom under the 100-connection database limit. Use connectionTimeout of 1 s, validationTimeout of 1 s, and a query timeout of 2-3 s so a stuck query cannot hold a connection for 30 s. Disable automatic retries for non-idempotent checkout writes; allow at most one retry for idempotent reads with jitter, and enforce a retry budget of about 10%. Add a circuit breaker that opens when the error rate exceeds 50% over a short window. Validate with a load test at 200 requests/s and 10% injected database errors: require p99 below 500 ms, pool wait time near zero, active connections below 80% of the database limit, and no retry storm. Monitor pool wait time, retry rate, database connections and error budget in production before restoring normal traffic.