Holiday Sale: The Two-Hour Pass and the Fourteen-Hour Risk
A travel booking platform passes a 2-hour load test at 1,500 concurrent users with p95 1.8 s and no errors. The site is expected to run at that load for 14 hours during a holiday sale. After the short test, the operations team sees a slow memory increase on the booking service: heap usage rises from 1.2 GB to 1.9 GB over 2 hours, and full GC pauses increase from 150 ms to 900 ms. CPU stays at 65%. The application uses a connection pool of 100 connections and a 30-second database timeout. The test environment has the same heap size as production but only one-third of the database CPU. The business asks whether the platform is ready. No errors were seen, but throughput was flat at 42 bookings/second.
1. What test type is missing, and why does a 2-hour pass not establish readiness for a 14-hour sale?
Read the worked answer
The missing test type is an endurance or soak test: a sustained steady-state run long enough to expose time-dependent degradation. A 2-hour load test mainly proves short-term capacity and stability. It does not prove that the system can hold the same p95 and error rate for 14 hours.
The evidence already shows a time-dependent trend: heap rose from 1.2 GB to 1.9 GB in 2 hours, and full GC pauses rose from 150 ms to 900 ms. If the heap growth is roughly linear, that is about 350 MB per hour; over 14 hours it would add about 4.9 GB. Starting from 1.2 GB, that approaches or exceeds a typical heap and would cause long or frequent full GCs, timeouts, or out-of-memory errors. The flat 42 bookings/second does not rule this out, because throughput can remain stable until GC or pool exhaustion suddenly dominates.
Endurance testing also checks connection-pool leaks, thread leaks, cache growth, log and file accumulation, and scheduled-job interference. The team needs a run longer than the sale window, for example 16 to 24 hours, at the expected 1,500-user load, with monitoring of heap after GC, GC pauses, thread counts, connection pools, and response-time drift. A 2-hour pass is a green light for short-term load, not for a 14-hour sale.
2. Interpret the heap, GC, CPU, and database data. What additional metrics would you require, and what would make the endurance test fail?
Read the worked answer
The CPU at 65% suggests the application is not CPU-bound, so the rising heap and GC pauses are the more important signals. Heap growth from 1.2 GB to 1.9 GB in 2 hours may be a leak or simply a growing cache; the key metric is heap used after full GC, not total heap. If heap after GC keeps rising, memory is being retained. Full GC pauses rising from 150 ms to 900 ms threaten the 30-second database timeout and the p95 SLA, because long pauses stop request processing and can cause connection-pool waits.
Additional metrics: heap after full GC, GC frequency and pause distribution, allocation rate, thread count, connection-pool active/idle/wait counts, database sessions and query latency, response time by transaction, throughput, error and timeout counts, queue lengths, and network errors. Because the test database has only one-third of production CPU, database latency may be pessimistic or may hide application issues; report it as an environment limitation.
Fail criteria should include: heap after full GC trending upward across the run; GC pause above an agreed threshold such as 500 ms; p95 drifting more than 20% above the first-hour baseline; error rate rising above 0.1%; connection-pool exhaustion or waits; throughput declining; and any out-of-memory or forced restart. A single spike is not enough; the trend and correlation with response time matter.
3. Design the endurance execution and analysis plan, including how to avoid a false pass or false fail given the database capacity difference.
Read the worked answer
Run at the expected 1,500 concurrent users for at least 14 hours, preferably 16 to 24 hours, with a ramp-up, a long steady state, and a controlled ramp-down. Use production-like data volume and session mix. Hold the load steady; an endurance test should not gradually increase load, or it becomes a stress test and confuses the analysis.
Before the run, capture a baseline: heap after full GC, GC counts and pauses, thread count, pool size, database latency, and p95 at the start of steady state. During the run, monitor at short intervals: heap after GC, GC pause distribution, pool active/idle/wait, database sessions, CPU, response time by transaction, throughput, errors, and timeouts. Take a heap dump if heap after GC rises consistently. Compare the last hour with the first hour of steady state, not with the ramp-up.
The database has one-third of production CPU. That may make the test pessimistic for database-bound paths, which could cause a false fail. It may also mask application inefficiencies if the smaller database is faster due to cache effects. Avoid a false pass by not declaring database readiness from this environment; run a separate database-capacity test or repeat at production scale. Avoid a false fail by separating application-tier metrics from database wait metrics and by reporting the database limitation. Use production APM during a canary period as the final confirmation.