Load-Test Verdict When Results Are Inconsistent and the Environment Differs
An independent test laboratory is running a load test for a retailer's checkout service. The service-level agreement states p95 response time of 800 ms or less and error rate of 0.5% or less at 5,000 concurrent users. The laboratory executes eight runs of the same script. Six runs show p95 between 610 and 740 ms with error rates below 0.3%. Two runs show p95 of 1,240 ms and 1,310 ms, with error rates of 3.1% and 2.8%; both failing runs show timeouts at the payment gateway. The laboratory environment uses 6 application servers and one database read replica, while production uses 10 application servers and three read replicas. The laboratory database holds about 20% of production data volume. The test script uses a fixed 2-second think time. The client wants the laboratory to declare a pass based on six of eight runs, citing 'normal variance', and points out that the go-live date is fixed. The laboratory's accreditation requires it to report valid results and to state limitations.
1. What verdict should the laboratory issue, and how should the report be worded?
Read the worked answer
The laboratory should not declare a pass. A pass would assert that the service met the service-level agreement under the tested conditions, but two of eight runs breached both thresholds. A 25% observed failure rate is not 'normal variance' unless a statistical basis is provided, and the failing runs are not random noise because both show the same symptom: payment gateway timeouts. The correct output is a failure in the tested configuration, combined with a clear statement that production compliance is not established because the environment is not equivalent. Precise wording matters: 'The checkout service did not meet the stated p95 and error-rate criteria in the laboratory configuration on 2 of 8 runs; the laboratory configuration differs from production in application-server count, read-replica count and data volume; therefore these results cannot be used to certify production service-level compliance.' Avoid 'pass with exceptions' or 'pass on majority', which would misrepresent the two breaches. The report should include the failing run timestamps, error logs and the gateway timeout evidence so the result is traceable.
2. What changes would you require in the test design before a result could support a service-level statement?
Read the worked answer
First, address environment parity or quantify the gap. Rerun with 10 application servers and three read replicas, and load production-scale data or a documented representative subset; if that is impossible, run a calibration test at reduced load to model scaling and report the limitation explicitly. Second, replace the fixed 2-second think time with a randomized distribution and separate user journeys, because fixed pacing can synchronize requests and create artificial spikes or hide queueing behavior. Third, isolate the payment gateway with a stub of known latency, then rerun against the real gateway to separate gateway behavior from application behavior. Fourth, predefine pass/fail and stop rules. Fifth, increase repetitions: with 2 failures in 8 runs, the uncertainty around the failure probability is wide, so more runs are needed before claiming stability. The laboratory should also validate the load generator's capacity so that generator saturation is not mistaken for application failure.
3. How should the laboratory handle the client's request to use the majority result, including escalation and impartiality?
Read the worked answer
The laboratory should decline the request in writing and document it. The accreditation requirement to report valid results and the rules on statements of conformity do not allow a majority vote when two runs breached the agreed thresholds. The laboratory can offer alternatives: a factual report without a conformity statement, a statement that production behavior was not evaluated, or a retest under agreed parity conditions. It should not let the go-live date determine the verdict. If the client insists, escalate to the quality manager and, if necessary, to the accreditation body's technical contact; the laboratory can withdraw from issuing a conformity statement rather than issue a misleading one. The final report should include the limitations, environment differences, the two failing runs and a clear conclusion. This protects impartiality and avoids a statement that the client could later use to claim production readiness that the evidence does not support.