Reading load, stress and soak results for a booking API

"Respond quickly at peak" cannot pass or fail, so it must be converted into a measurable, condition-bound requirement agreed with the business owner. A testable form: "With the agreed workload mix sustained for 60 minutes at 700 requests/second, on the production-equivalent environment with a production-sized data set, the search endpoint shall return 95% of responses within 1.0 s, 99% within 2.5 s, with an error rate below 0.5%, and no application-tier host above 80% CPU."

That version fixes the elements that were previously implicit: metric (percentile, not average or median), threshold, workload mix and rate, duration, environment and data volume, and error tolerance. The 1.0 s threshold must come from the business or from a contractual SLA, not from the current measurement, otherwise the requirement merely describes existing behaviour.

Before measuring, pin down the workload model (read/write ratio, search parameter distribution, think times, session length), the environment (hardware, network, JVM settings, data size), and the monitoring that will be used. Also decide in advance which percentiles and error definitions matter: a 400 ms median alongside a 4.2 s 99th percentile means roughly one in a hundred users has a materially bad experience, which the median conceals. Record the tooling and the load generator's own capacity so that a plateau is not mistaken for a client-side limit.

The evidence points to a saturated application tier, not the database. Throughput flattens at roughly 700 rps while application CPU is at 96% and database CPU only 40%, so adding load cannot produce more work: requests queue, latency grows, and eventually timeouts produce the 7% error rate. The virtual-user increase from 500 to 900 confirms this - more concurrency, same throughput, worse latency - which is the classic signature of a capacity ceiling rather than a load-generation problem.

Likely causes within the application tier: garbage collection pressure, thread-pool or connection-pool exhaustion, lock contention, or expensive serialisation. These should be separated with instrumentation (GC logs, thread dumps, profiler) during a stepped test - for example, ramping 300 → 500 → 700 → 900 rps in fixed stages and correlating each stage with CPU, GC time and pool occupancy.

Follow-on tests:

• Stress test beyond the ceiling, deliberately, to observe degradation and confirm recovery after the load is removed - does the system return to baseline or stay degraded?
• Scalability test: add an application instance and re-measure, to check whether throughput scales horizontally; that distinguishes a code-level bottleneck from a capacity shortfall.
• The wide 95th/99th spread should be investigated separately, since tail latency often indicates contention or a minority path such as a cache miss.

Note that the plateau may be acceptable if 700 rps exceeds the forecast peak - but that must be an explicit decision against the requirement, not an accident.

A heap that grows from 1.1 GB to 1.9 GB over 12 hours and is never reclaimed by full GC is a memory leak, or at least unbounded growth. The absence of an OutOfMemoryError is not reassurance: the test simply ended before the heap filled. Extrapolating the 800 MB growth, the service reaches any plausible maximum heap within a few days of continuous operation - so the symptom in production would be a crash or forced restart, not a slow response. That is a reliability defect, and it belongs under non-functional testing even though it appears in a performance test.

Actions before release:

1. Root-cause it: rerun the soak with heap dumps taken at intervals and analyse the dominator tree to find the retained objects; capture GC logs throughout.
2. Extend the soak duration and run it at the target peak rate rather than 300 rps, since leak rate often scales with traffic.
3. Check whether the leak is a known growth pattern (caches without eviction, growing session or metrics maps, unbounded queues) rather than an exotic bug; these are usually fixable quickly.
4. Do not accept "raise the heap" as a fix - it only lengthens the interval to failure. If the fix cannot land in ten days, mitigate explicitly: scheduled restarts, heap-usage alerting with a threshold well below the maximum, and a documented residual risk agreed with the product owner.
5. Add a recovery test: after a restart or a crash, does the service resume correctly, and is in-flight work handled? This should be part of the release criteria alongside the soak test.