Checkout Load Test: SLO Miss Before Peak
A retail checkout service is preparing for a peak event expected at three times normal traffic. A 90-minute load test at that target, with a 15-minute ramp, produced p95 checkout latency of 1.8 seconds against a 1.2-second SLO and an error rate of 0.4% against a 0.1% target. Application CPU averaged 55%, database CPU reached 92%, and one of four read replicas was saturated. The test used a 20% skew in customer IDs and a cold cache. A test at twice normal traffic passed one week earlier. The release adds a database index but also introduces a new recommendation call on the checkout path. No production-like warm-up was performed. The business wants sign-off today; the next test window is after the peak.
1. Determine whether the failed SLO is a product defect, a test artifact, or both. What evidence separates them?
Read the worked answer
The failure is probably both a product-capacity issue and a test artifact, and the decision needs evidence to separate them. Database CPU at 92% and one saturated read replica point to a database or read-path bottleneck; application CPU at 55% argues against an application compute limit. The cold cache and 20% customer-ID skew can exaggerate latency and error rate, so a warm-cache rerun with production-like skew is needed. The new recommendation call adds read load on the checkout path, and the new index may help only specific queries. Separate the signals by collecting per-endpoint latency, slow-query logs, connection-pool saturation, replica lag, and error codes; then rerun at three times peak with warm cache and realistic skew. If p95 remains above 1.2 seconds and errors remain above 0.1% when the recommendation call is disabled, the bottleneck is in the baseline checkout path, not the new feature. If disabling the recommendation call restores SLO, the feature is the trigger and should be gated.
2. Recommend release or no-release, and specify the test or mitigation needed before sign-off.
Read the worked answer
Do not sign off on the current evidence. A conditional release is defensible only if the recommendation call can be feature-flagged off at runtime, the read replica is scaled or the query is fixed, the index is verified with production-like data, and a warm-cache test at three times peak meets the SLO or the business accepts a documented error budget. Otherwise recommend no-release for the peak. The next test should include a warm-up, production-like skew, and a control run with the feature disabled so the team can attribute the gap. If the test window is after the peak, use a canary with automatic rollback on p95 or error-rate breach. The product owner should see the residual risk in terms of customer-visible failures, not just CPU charts.
3. What would you change about the test design for the next window?
Read the worked answer
Redesign the test to answer attribution questions: run a baseline at three times peak with the feature disabled, then with it enabled; warm the cache before measurement; use production-derived customer-ID skew and cart mixes; run long enough to expose connection-pool and replica-lag behavior; monitor database and replica metrics at the same granularity as latency; and include failure-mode tests such as one replica down. Also test at two times peak as a control and define pass/fail per endpoint, not only aggregate p95. The next window should be scheduled before the peak, with a rollback criterion agreed in advance.