Integration server overload: JDBC pool waiters, heap pressure, and MQ backlog
An ACE V12 integration server runs on a four-vCPU, 8 GB Linux virtual machine with maxHeap set to 2 GB. It hosts 150 message flows and handles about 200 concurrent HTTP requests. Each request performs five JDBC calls through a data source whose maximum pool size is 20. Peak traffic produces p95 latency of 8 seconds, CPU at 70 percent, heap at 85 percent after full GC, and about 40 request threads waiting for a JDBC connection. The database shows 100 percent connection utilization and some idle-in-transaction sessions. IBM MQ backlogs grow during peaks and drain slowly. The team has already restarted the integration server twice; each restart clears the backlog temporarily, then the symptoms return within an hour. A proposal is to raise maxHeap to 4 GB and double the HTTP thread pool.
1. Using the symptoms, identify the primary bottleneck and the first diagnostics you would run before changing configuration.
Read the worked answer
The strongest signal is 40 threads waiting for a JDBC connection while the pool maximum is 20. That is pool exhaustion, not proof of heap exhaustion. Heap at 85 percent after full GC indicates a high live set, but the waiters explain the latency and the MQ backlog more directly. CPU at 70 percent means the JVM is not CPU-saturated; it is spending time waiting.
Run three diagnostics in parallel. First, take two or three thread dumps about 10 seconds apart and count threads blocked in connection-pool wait versus socket reads or GC. Second, collect integration server resource statistics and data-source pool statistics: wait time, in-use connections, connection leaks, and transaction duration. Third, inspect the database for long-running statements, idle-in-transaction sessions, and max_connections headroom. Also check whether flows close connections on every exception path; a leak would explain why a restart helps only temporarily.
Only after those results should you change maxHeap or thread counts. Doubling HTTP threads when the pool is already exhausted increases queueing and can worsen the database load.
2. Propose a configuration and design change set, with trade-offs. Include at least one change that is not simply increasing a pool size.
Read the worked answer
First, right-size the JDBC pool against the database and the request concurrency, not against the number of HTTP threads. If the database can safely allow 60 connections for this server, raise the pool to 60 and add a connection-validation query and a leak-detection timeout. If the database cannot, reduce HTTP concurrency instead and add backpressure.
Second, reduce connection hold time. Move database work out of long-running flows, batch calls, and make sure each transaction commits or rolls back promptly. The idle-in-transaction sessions suggest connections are held while waiting on something else. Set a statement timeout and a transaction timeout.
Third, isolate workloads. Split the 150 flows into two or three integration servers by latency and criticality, each with its own heap and pool, so one heavy flow cannot consume all connections. This costs more JVMs and operational complexity but limits blast radius.
Raising maxHeap to 4 GB may be reasonable if the live set is genuinely large, but on an 8 GB host it leaves less room for OS page cache and other processes. Bound the HTTP thread pool to a tested value rather than doubling it blindly.
3. How would you apply these changes with minimal disruption and verify that the fix is real, not just a restart effect?
Read the worked answer
Use a rolling or blue-green deployment rather than an in-place restart of the only server. Build the configuration changes into server.conf.yaml and the BAR deployment, deploy to a second integration server, run a canary or synthetic load against it, then move traffic at the load balancer. Keep the old server available for rollback until the new one has survived at least one full peak period.
Verify with leading indicators, not only queue depth. Track JDBC pool wait time, active connections, transaction duration, HTTP p95, heap after full GC, GC pause time, and MQ backlog age. A fix is credible only if pool wait time drops and p95 stays within target at the same offered load, without a restart. If the backlog drains only after a restart, the underlying pool or transaction behavior is still wrong.
Finally, add alerting on pool wait time and idle-in-transaction sessions. Do not treat MQ backlog as the root cause; it is a downstream symptom of the same saturation.