Subscription API: Averages, Percentiles, and Little's Law
After a load test of a subscription API, the report states: 500 virtual users, 60-minute steady state, average response time 450 ms, throughput 900 requests/minute, error rate 1.0%, CPU 78%. The business sponsor rejects the result because customer complaints mention slow checkout. A test engineer notes that the load script uses a uniform 5-second think time, all users start with login, and only the average is reported. Raw data shows p50 = 320 ms, p95 = 2.8 s, p99 = 6.4 s, and errors concentrated in the payment authorization call. The workload mix was 80% login and 20% payment, while production is 20% login and 80% payment. The SLA is p95 < 1.5 s and error rate < 0.5%. The API is stateless behind a load balancer and the payment service calls an external gateway.
1. Check whether the reported 500 virtual users, 5-second think time, 900 requests/minute, and 450 ms average response time are internally consistent. Show the calculation and state what the inconsistency implies.
Read the worked answer
Use Little's Law for a closed workload: N = X × (R + Z), where N is concurrent users, X is throughput, R is response time, and Z is think time.
- X = 900 / 60 = 15 requests/second.
- R = 0.45 s; Z = 5 s.
- N = 15 × (0.45 + 5) = 15 × 5.45 = 81.75 users.
If think time is excluded, N = 15 × 0.45 = 6.75 users. Neither result is close to 500.
The numbers are not consistent. With 500 users, a 5-second think time, and a 0.45 s response time, the theoretical maximum throughput would be 500 / 5.45 ≈ 91.7 requests/second, or about 5,500 requests/minute. Conversely, if the server really handled only 15 requests/second with 500 users and 5-second think time, response time would be R = 500/15 − 5 ≈ 28.3 seconds, not 450 ms.
Likely causes include: think time not applied in the script; 500 users configured but not all active; throughput measured on only one transaction type; the average computed across a mix with very different response times; or ramp-up/ramp-down included in the average. The implication is that capacity conclusions drawn from these figures are unsafe. The team must reconcile raw logs, arrival model, active user count, and per-transaction response times before reporting again.
2. Interpret p50, p95, p99, and the error concentration. What should the report say about SLA compliance?
Read the worked answer
The percentiles tell a different story from the average:
- p50 = 320 ms means half the requests are fast, which explains why the average looks acceptable.
- p95 = 2.8 s exceeds the SLA of p95 < 1.5 s. By the stated criterion, the test fails.
- p99 = 6.4 s shows a long tail. Even if the p95 were borderline, users would experience multi-second waits regularly enough to generate complaints.
- Error rate 1.0% exceeds the SLA of < 0.5%. Errors concentrated in payment authorization point to a specific integration or timeout problem, not random noise.
The average of 450 ms should not be used as the headline result. It hides the tail and is inconsistent with the SLA. The report should state: SLA not met on p95 and error rate; the payment authorization path is the main risk; login performance may be acceptable but is overrepresented in the mix. Because the test used 80% login and 20% payment while production is 20% login and 80% payment, the true production-weighted response time is likely worse than the blended test figure. Report per-transaction percentiles, error rates by endpoint, throughput stability, and the workload mix. Avoid a single 'pass/fail' based on average.
3. Propose changes to the workload model and test design before rerunning. Include at least one way to avoid the payment gateway becoming an unrepresentative bottleneck.
Read the worked answer
The rerun should model production, not an arbitrary script mix.
- Use production transaction weights: 20% login and 80% payment, or better, derive weights from production logs over the busiest hour. Keep separate transaction definitions and separate SLAs; do not blend login and payment into one average.
- Use realistic arrival and think times: model the session arrival pattern and user think time from production. Size virtual users with Little's Law and verify active concurrency, rather than setting 500 users and assuming they are all active.
- Parameterize and separate data: unique users and payment tokens; avoid cached login sessions that make login unrealistically fast.
- Measure per endpoint: p50, p95, p99, error rate, throughput, and timeouts for login and payment authorization separately.
To avoid an unrepresentative payment bottleneck, use service virtualization or a controlled stub for the external gateway during load tests, with latency and error behavior matching production. Then run a separate integration test at the gateway's real capacity. Alternatively, obtain a production-like gateway sandbox with sufficient throughput. Monitor gateway latency and error responses as a first-class metric. If the gateway is the bottleneck, report it as an external dependency risk rather than attributing the failure to the API.