AWS Solutions Architect study materials
AWS Solutions Architect study resources

The 09:00 Connection Storm

AWS Solutions Architect · Put your knowledge to work in context.

All case studies

The 09:00 Connection Storm

A B2B analytics SaaS runs 40 m6i.2xlarge application instances behind an ALB, each with a HikariCP pool of 25 connections to a single db.r6g.4xlarge Aurora PostgreSQL writer. Between 08:55 and 09:20 UTC, p95 API latency rises from 300 ms to 9 s and the ALB returns 12% 5xx. CloudWatch shows DatabaseConnections pegged at the writer's connection limit, CPU at 88%, and roughly 3,000 connection attempts per minute failing with 'too many clients'. Autoscaling is target tracking on CPU at 60%, which adds instances during the incident. Two reporting jobs start at 09:00 and each hold a single connection for 20 minutes. The team's first instinct is to double the instance to db.r6g.8xlarge.

1. Explain why both proposed fixes (bigger instance and more instances) make the situation worse, then rank the actual remedies.

Read the worked answer

Why the instinct is wrong. HikariCP defaults minimumIdle to maximumPoolSize, so each instance eagerly opens 25 connections at boot. Forty instances already represent 1,000 potential connections - the pool arithmetic, not the query load, is what the writer sees. Autoscaling to 60 instances raises the ceiling to 1,500. Doubling the instance buys headroom proportional to memory, but every PostgreSQL backend costs memory and scheduler time whether or not it is running a query, so you pay 2x to delay the same wall by minutes.

Ranked remedies:

  1. Break the incident (minutes). Drop each pool's maximumPoolSize to 8–10 via a config push and recycling, and cap ASG max so the fleet cannot multiply its way into the limit. Move the two 09:00 reporting jobs to a reader or reschedule them out of the peak window - two 20-minute transactions holding connections are cheap to fix and immediate.
  2. Fix the scaling signal (days). Target tracking on CPU is targeting the wrong variable: when latency comes from DB waits, application CPU stays moderate while requests queue. Scale on ALB RequestCountPerTarget or TargetResponseTime instead, and set a scale-in cooldown longer than the query duration so the fleet does not flap.
  3. Decouple connection count from instance count (weeks). Put RDS Proxy or PgBouncer in transaction-pooling mode in front of the writer. Critical caveat: RDS Proxy pins a session whenever it sees prepared statements, SET, or an open transaction, and a pinned session multiplexes for nobody. Audit the driver configuration (server-side prepared statements, prepareThreshold) before assuming you get pooling, and measure the pinning rate after rollout.

Order matters: proxy first, pool tuning second, and you will never learn whether the proxy helped. Tune the pool first, then let the proxy absorb the remaining churn.

2. Design the read path for reporting and state the consistency trade-off, then describe how you would prove in a load test that the change holds before the next month-end.

Read the worked answer

Read path. Give reporting its own Aurora reader instance behind the cluster reader endpoint (the reader endpoint only routes to Aurora Replicas, so a dedicated replica is required - with no replicas it silently falls back to the writer and you are back where you started). Reporting gets its own pool sized to its concurrency, typically 4–8, and its own connection budget so a runaway report cannot starve the API.

Consistency trade-off. Replicas are asynchronous; lag is normally tens of milliseconds but spikes under write bursts. Reporting that reads financial aggregates immediately after a write can be wrong. Decide per report: tolerate bounded staleness with a displayed 'as of' timestamp, or read the writer for the handful of reports that must be exact and accept that they are slow. Do not paper over this with a retry loop - it converts a visible staleness bug into an invisible latency bug. Alarm on AuroraReplicaLag and fail the report if lag exceeds the tolerance rather than serving stale money.

Proving it. Replay a captured production request mix at 1.5x peak using a load generator, and run the two reporting jobs on their real schedule inside the window - a load test without them proves nothing about this incident. Assert on: writer DatabaseConnections staying under 60% of the limit, p95 TargetResponseTime under 500 ms, zero 'too many clients' errors, and ASG scale-in not oscillating. Run the test twice: once with the pool changes only, once with the proxy added, so you can attribute the improvement and know which control actually saved you.