400 GKE Pods, 6,000 Idle Connections, and an RTO Nobody Has Tested

'Too many connections' at 35% CPU is a connection-management failure, not a capacity failure. 400 pods x 15 connections = 6,000 concurrent backend connections. PostgreSQL allocates per-connection memory (roughly 5-15 MB of backend state plus work_mem for active queries), so 6,000 direct connections would consume tens of gigabytes before running a single query, and Cloud SQL's default max_connections for that tier is in the hundreds - raising the flag just moves the failure to memory exhaustion.

Architecture: a pooling layer between pods and the database.

• Deploy PgBouncer (or AlloyDB/Cloud SQL's managed pooling if you move) in transaction pooling mode as a sidecar or as a dedicated Deployment behind an internal Service, sized to ~1,000 server-side connections total across the primary and replicas. Transaction pooling works here because the app's 92% reads are short; it does not work for session-state features (prepared statements with session scope, advisory locks, LISTEN/NOTIFY) - audit for those first, they are the usual reason pooling 'breaks' an application.
• Use the Cloud SQL Auth Proxy/connector on the client side for IAM-based auth and TLS, not for pooling - the proxy multiplexes certificates, not connections.
• Set max_connections on the instance to a deliberate number based on memory, leave headroom for admin and replication, and alert on connection utilization above 70% so this never surprises anyone again.

On the 90-second maintenance failure: Enterprise Plus instances are designed for near-zero-downtime maintenance, but the application must still tolerate the moment a connection is invalidated. Configure the pooler to retry and to validate connections, set short connect timeouts, and add retry-with-backoff at the application layer for idempotent operations. Without that, any failover - planned or unplanned - produces user-visible errors and gets misattributed to the platform.

Expected effect: server-side connection count drops by roughly 80%, memory pressure disappears, and p95 read latency improves because the database is no longer spending time on connection setup. No new spend beyond a small pooler deployment.

Topology: Cloud SQL Enterprise Plus primary in us-central1 with a cross-region replica in us-east1, promoted by a documented, scripted runbook. A cross-region replica gives asynchronous replication with typical lag in the low seconds; RPO 5 minutes means the design must tolerate replication lag up to 5 minutes, so monitor and alert on replica lag as the RPO SLI - if lag ever exceeds 5 minutes, you are outside your stated RPO and the on-call should know before an incident, not during one.

RTO 10 minutes is achievable only if the promotion is rehearsed. Promotion itself takes a few minutes; DNS/proxy repointing and connection re-establishment take the rest. Budget roughly: 1-2 minutes to declare and page, 2-4 minutes for promotion, 1-2 minutes for the pooler and pods to reconnect, 1-2 minutes of verification. That leaves no slack for improvisation, so the runbook must be scripted and the GKE services must read the database endpoint from a configuration source that can be updated once, not baked into 400 pod specs.

Why not a hot standby read-write instance on both sides: an active-active PostgreSQL pair in two regions requires conflict resolution the application does not have; for this workload it is complexity without benefit and would break the budget.

Budget: a cross-region replica roughly doubles database spend. To stay inside 25%, move the read-heavy 92% onto the replica path in the primary region where possible, right-size the replica, and let the pooler absorb concurrency instead of buying a larger tier. Itemize this rather than asserting it.

Proving the numbers: run a quarterly unannounced drill that promotes the replica, measures actual wall-clock RTO to a successful write from the application, and records data loss against the replication position. A DR plan with untested numbers is a hope. Post-drill, capture the measured RTO/RPO and the exact runbook version in the incident system.

First, quantify before migrating. 4,200 queries/s at 35% CPU suggests headroom today; the question is what breaks first as reads grow. Cloud SQL read replicas scale reads linearly and are operationally simple: add a replica, add it to the pooler's read pool. The real cost is replica lag, which is unacceptable for read-your-writes screens (an order just placed must appear immediately). Check how many of the 4,200 reads/s require read-your-writes - typically a small fraction, and those must be pinned to the primary.

Cloud SQL with replicas - when it wins: read-heavy, latency-tolerant traffic; a team that wants one operational model; a workload that stays under a few tens of thousands of reads/s. Cost is linear in replicas, and each replica needs its own pooling configuration.

AlloyDB read pool - when it wins: the read pool scales reads without you managing individual replicas, and it also serves the analytical queries that would otherwise force a separate BigQuery copy. If the 40% growth includes reporting scans against the same tables, the columnar engine changes the economics materially.

Recommendation: a staged path. (1) Immediately add a second read replica and route the read-only pool to replicas through the pooler, keeping read-your-writes on the primary via a per-request routing flag. (2) Instrument query-level cost to find whether growth is many small reads or a few expensive scans. (3) If it is a few expensive scans, that is an AlloyDB or BigQuery case, not a replica case - replicas only compound cost. (4) Re-evaluate the AlloyDB move when the primary sustains above ~60% CPU at peak or when replica count exceeds three.

Migration caution: moving the read path to AlloyDB while writes stay in Cloud SQL creates a replication dependency and a partial-failure mode. If you move, move the whole instance - do not run a split-brain half-migration for a read-scaling problem.