Checkout Latency at 1,200 Virtual Users Before a Flash Sale
TicketWave sells concert tickets. During a load test with 1,200 virtual users over 30 minutes, the checkout API's 95th percentile response time rose from 600 ms to 4.2 seconds, error rate reached 8%, and CPU on the primary PostgreSQL server stayed at 55%. Application logs show HikariCP connection-acquisition timeouts after 12 minutes and frequent full garbage collections on two Java nodes. The front-end static assets are served by a CDN and remain fast. The team has JMeter, Grafana, Prometheus, and Java flight recorder. The product owner wants to add more application nodes. The architect suspects a missing database index. There is one week before a flash sale expected to bring 3,000 concurrent users. The non-functional requirement is p95 < 1.5 s and error rate < 0.5% at 3,000 users.
1. Plan a technical investigation to isolate the bottleneck. Specify hypotheses, experiments, and the metrics or evidence you would collect.
Read the worked answer
Do not add nodes first: more application nodes may open more database connections and worsen contention. Form competing hypotheses and test them with controlled experiments.
Hypotheses
- Slow queries cause connection-pool exhaustion: pool waiters time out while database CPU is only 55%, which suggests waiting or inefficient queries rather than saturation.
- Garbage collection pauses on the two Java nodes stretch request times and hold connections longer.
- A missing index drives excessive row scans or lock waits in checkout.
Experiments and evidence
- Run a stepped load profile (300, 600, 1,200 users) and capture p95, throughput, error rate, HikariCP active/idle/wait metrics, JVM heap and GC logs, and database wait events, locks, and slow-query statistics.
- Use Java flight recorder on one node to see whether time is spent in GC, connection acquisition, or query execution.
- Test the suspected index on a production-like clone with realistic data volume; compare query plans and execution time.
- Vary pool size in a controlled run to see whether errors fall or move downstream.
- Check whether checkout contention is caused by test data (for example, all users targeting the same seats), which would be a test-design artefact.
The aim is to separate application, connection-pool, garbage-collection, and database causes before choosing a fix.
2. Design the performance test scenarios and acceptance criteria for the flash sale, including workload model and test data considerations.
Read the worked answer
Model a realistic mix rather than sending every virtual user to checkout. For example: 60% browse/search, 25% view event or seat map, 15% checkout, with think times and a ramp to 3,000 concurrent users. Add a spike scenario that jumps from 500 to 3,000 users, and a shorter endurance run to expose connection leaks or GC degradation.
Use unique users and realistic seat/event data, but deliberately include a small percentage of contention cases for popular events. Avoid making every user buy the same seat; that measures database locking rather than normal capacity. Separate checkout and browse results so a fast CDN does not hide the checkout problem.
Acceptance criteria at 3,000 users: checkout p95 below 1.5 seconds, overall error rate below 0.5%, and no connection-acquisition timeouts. Also monitor business outcomes such as completed orders, database wait events, GC pause time, and pool saturation. Define a stop condition so the test aborts if error rate exceeds a safety threshold, and retain evidence for comparison after fixes.
3. How would you report the findings and prioritise fixes with one week remaining, given uncertainty about the root cause?
Read the worked answer
Report in risk terms: current evidence shows the system fails the requirement at 1,200 users, let alone 3,000, and the failure is in checkout rather than static content. State confidence levels: connection-pool timeouts and GC are observed; the missing index is still a hypothesis until query plans and clone tests confirm it.
Prioritise by expected risk reduction per unit of effort. Quick, reversible checks include pool sizing, query review and index addition on a clone, GC tuning, and reducing connection pressure from non-checkout paths. In parallel, prepare a capacity plan and a fallback such as queueing or load shedding for the flash sale. Do not treat a successful retest as proof unless it uses production-like data volume and a realistic workload. Report residual risk explicitly, including what remains unknown and what monitoring will trigger during the sale.