Connection Pool Pressure in a 60-Pod Service
A Java service runs on 60 Kubernetes pods and connects with Neo4j Driver 5.18 to a Neo4j 5.20 Enterprise cluster with 3 primaries and 6 secondaries. Each pod configures maxConnectionPoolSize=100 and connectionAcquisitionTimeout=5s. The driver uses the neo4j:// scheme. Every HTTP request opens a session, runs one or more queries, and closes the session in most, but not all, code paths. Some reads call session.run() and return the Result without consuming it; a few write paths use autocommit session.run() inside a loop. Under load, p99 latency reaches 12 seconds, pods log failures to obtain a connection from the pool within 5 seconds, and the cluster reports about 4,800 active connections from the application. The team wants to keep 60 pods but reduce connection pressure and route reads away from the primaries.
1. Calculate the theoretical worst-case connection footprint and explain why the observed 4,800 is plausible. What pool size would you test, and what else must change?
Read the worked answer
The driver maintains a pool per server address, and maxConnectionPoolSize is per pool. With 9 cluster members, 60 pods, and 100 connections, worst case is 60 x 100 x 9 = 54,000 connections, although connections are created lazily and only to servers used by routing. 4,800 is consistent with partial pool growth and leaks. Reducing maxConnectionPoolSize to 20 gives a ceiling of 60 x 20 x 9 = 10,800, still large but more manageable; 10 gives 5,400. Pool size is not the root cause if sessions or results are not closed. Set a smaller pool, keep connectionAcquisitionTimeout, and fix lifecycle leaks. Monitor dbms.listConnections and driver metrics; do not size only by pod count.
2. Rewrite the read and write patterns to avoid leaks and to use managed transactions. Include session closure and result consumption.
Read the worked answer
Use try-with-resources for sessions and executeRead/executeWrite for transaction functions. Do not return a Result from a method after the session closes. Consume with list(), consume(), or iterate inside the callback.
try (Session session = driver.session()) {
List<Record> rows = session.executeRead(tx ->
tx.run('MATCH (u:User {id:$id}) RETURN u.name AS name', parameters('id', id))
.list()
);
}
try (Session session = driver.session()) {
session.executeWrite(tx -> {
tx.run('MERGE (u:User {id:$id}) SET u.name=$name', parameters('id', id, 'name', name))
.consume();
return null;
});
}
For a loop of writes, run one transaction function with a batch, not one autocommit query per iteration. Always close sessions in finally or try-with-resources; a session that is garbage-collected later can hold connections.
3. Why might reads still hit primaries, and how would you verify read routing after the change?
Read the worked answer
session.run() without an access mode defaults to write and is routed to a primary. executeRead sets READ access, so the driver's routing table can choose a secondary; if no secondary is available, reads may still be served by a primary. The neo4j:// scheme enables routing; bolt:// does not. Verify by checking the routing table and server role, and by observing query distribution across cluster members. Use the routing procedure available in your version, or driver logs and metrics, and compare before and after. Also ensure the read query is actually read-only: a procedure with write effects or an explicit write transaction will force a primary. If secondaries lag, bookmarks and causal consistency still apply; reads after a write may need the primary.