HPA Scale-Out Meets a 100-Connection PostgreSQL Ceiling
A Java service on Red Hat OpenShift on IBM Cloud runs on six 16 vCPU/64 GB workers. The HPA is set to 8–40 pods on 70% CPU, and each pod uses HikariCP with a maximum pool of 10. The backing IBM Cloud Databases for PostgreSQL plan enforces 100 connections. During a marketing peak, traffic doubles; the HPA reaches 40 pods, the database logs remaining connection slots are reserved, p99 latency rises from 300 ms to 4 s, worker CPU reaches 82%, and database CPU is 95%. There is no read replica and no external pooler. The JVM heap is at 75% and GC time has tripled. The team proposes raising the HPA maximum to 80 and increasing the database plan to the next tier.
1. Diagnose the failure chain and identify which proposed change would help or hurt.
Read the worked answer
The failure chain is a connection amplification loop. Each pod can open 10 connections, so 40 pods can demand 400 connections against a 100-connection ceiling. Once the pool waits, request latency rises, CPU-based HPA sees more CPU from queued work and adds more pods, which increases connection demand further. Database CPU at 95% then slows every query, and JVM GC at tripled time reduces throughput per pod. Raising the HPA maximum to 80 is harmful: it would allow up to 800 connection attempts and worsen the queueing. Increasing the database plan may raise max_connections and CPU, but it is a capacity patch that does not fix the amplification or the CPU hotspot. It can be a temporary relief only if paired with a connection cap. The primary fixes are a pooler, a per-pod pool sized to the database budget, and HPA behavior that does not chase latency-induced CPU.
2. Give a sequenced remediation plan for the next 24 hours and the next two weeks.
Read the worked answer
Next 24 hours: cap the blast radius. Lower HikariCP maximum pool from 10 to 2–3 per pod and set the HPA maximum to a value that keeps total connections under 100 (for example, 30 pods at 3 connections). Add a connection pooler in front of PostgreSQL, or use the database service's supported pooling option if available, and route pods through it. Enable HPA scale-down stabilization and add a scale-up rate limit so a latency spike cannot trigger a pod stampede. Increase the database plan only as a stopgap and document the expiry. Next two weeks: load-test at 2x peak with the pooler and the new pool size; move reporting or read-heavy queries to a read replica; set JVM heap and GC tuning based on the test; add a capacity model that treats database connections as a first-class resource; and review CPU requests so the HPA scales on a signal that reflects work, not queueing. Reassess whether CPU is the right HPA metric or whether a custom metric such as requests per pod is better.
3. Define the capacity model and leading indicators you would put on the dashboard.
Read the worked answer
Capacity model: maximum safe pods = floor((database connection limit − reserved admin connections) / per-pod pool size). Track that against HPA maximum. Also track database CPU per 100 connections, query p99 by statement, pool wait time, and worker CPU headroom. Leading indicators: connection utilization as a percentage of the limit, pool acquisition wait time, HPA desired replicas vs available replicas, pending pods, JVM GC pause time, and database replication lag if a replica is added. Add an alert when connection utilization exceeds 70% for 10 minutes and when HPA desired replicas are within 10% of maximum. Do not rely on average CPU alone; a CPU-based HPA can be late and can overshoot when the real bottleneck is downstream. The dashboard should make the relationship between pods, connections, and database CPU visible on one screen so the next peak is a capacity decision, not a surprise.