Flash-sale 503s: connection exhaustion behind a connection-per-request handler

The scarce resource is connection slots, not CPU or query time.

Worst-case concurrent handlers = max-instances × concurrency = 300 × 80 = 24,000, each holding its own backend connection. The server admits 400. So the ceiling is ~1.7% of potential demand.

The failure is a positive feedback loop, which is why it looked like a latency problem:

1. A modest burst raises concurrency; instance count grows toward 300.
2. Connection demand crosses 400; new backends are refused instantly.
3. Handlers that do get a connection hold it longer (contention, locks), so latency rises.
4. Higher latency means Cloud Run keeps more instances busy for the same req/s → even more connections.
5. At the instance ceiling, requests queue until the 60 s timeout, and Cloud Run surfaces 503s (no capacity / saturated instances).

Why 35% CPU rules out the obvious alternatives: if the bottleneck were query cost or an undersized tier, CPU would be pinned near saturation. At 35% the database is mostly idle and refusing work - a quintessential admission-control failure. So scaling the tier, adding indexes, or rewriting queries would not fix it; they spend money on the wrong axis. The fix has to bound the number of concurrently open backends.

1. Replace the per-request connection with a per-instance pgxpool (highest leverage). Set MaxConns=8, MinConns=2, and a short pool.Acquire timeout (e.g., 500 ms). Connections now scale with instances, not with in-flight requests, so concurrency=80 no longer multiplies connection demand by 80. Cost: near zero; benefit: removes the 80x amplifier.

2. Cap max-instances so the pool math fits under max_connections. 40 instances × 8 connections = 320 open backends, leaving ~80 slots of headroom for migrations, admin sessions, and monitoring. Verify throughput before shipping the cap: 40 × 80 = 3,200 concurrent requests; at even 300 ms p99 downstream that is >10k req/s of capacity, far above the 900 req/s target. If the cap ever binds, raise max_connections (or the tier) rather than the cap blindly.

3. Set min-instances to cover baseline. min-instances=10 removes the 6 s cold start from the ramp and, because each instance opens its pool at startup, makes the steady-state connection count predictable instead of a surprise spike.

4. Add fail-fast instead of queueing. On pool-acquire timeout, return 503 with Retry-After in well under a second rather than letting the request sit for 60 s. Lower the request timeout to ~15 s. This converts a 30 s hang into a fast, retryable rejection and stops slow requests from pinning connections - the mechanism that turned a burst into a 30 s tail.

5. Evaluate Cloud SQL managed connection pooling for the instance if it is available in your region; it moves multiplexing server-side and lets you relax the instance cap. Treat it as an optimization, not the primary fix - steps 1–4 must be correct without it.

6. Add alerting on Cloud SQL num_backends / max_connections utilization at 70%, and on Cloud Run 503 rate, so the next sale is caught before users notice.

Reproduce the shape, not just the volume. The failure depends on ramp speed and steady-state concurrency, so the load profile must go 0 → 900 req/s in under 60 s and hold for 10 minutes. A single laptop cannot generate that reliably; use a distributed generator (Distributed Load Testing on GKE, or several regional VMs) so the client is never the bottleneck and does not distort tail latency.

Match the environment. Staging must use the same min-instances, max-instances, concurrency, timeout, and a Cloud SQL instance of the same tier and max_connections. Testing against a bigger database proves nothing.

Watch the right signals during the run:

• Cloud SQL: num_backends vs max_connections (must stay below the alert threshold), connection-refused count, CPU.
• Cloud Run: instance count curve, max concurrent requests, 503 rate, p99 latency.
• Application: pool-acquire wait time and acquire-timeout counts.

Deliberately test the pathological paths, because those are what actually broke last time:

• A step-ramp to failure (raise load past the instance cap) to confirm the system degrades into fast 503s rather than 30 s hangs, and recovers when load drops.
• A deliberately slow query or lock to confirm pool exhaustion fails fast and does not cascade.
• A scale-from-zero test to time the cold-start ramp with min-instances above zero.

Success criteria before the sale: p99 under target at 2× expected peak, zero connection-refused events, and a demonstrated recovery within seconds of load removal.