Rolling Out a Configuration Change Across Two Regions
A service runs in two regions, each with 12 instances behind a load balancer. A configuration change raises a connection pool limit from 20 to 60 per instance. The database allows 600 total connections. Currently 24 instances x 20 = 480 connections are in use at peak. The change is deployed by a rolling update that replaces instances one at a time per region, and both regions deploy simultaneously. Instances take 90 seconds to become healthy and hold their old connections for 30 seconds after the new process starts. There is no canary stage. A previous change of this type caused a 6-minute outage.
1. Compute the peak connection demand during the rollout and explain the outage risk.
Read the worked answer
During a simultaneous two-region rolling update, each replaced instance briefly runs old and new processes together. Old connections persist 30 seconds after the new process starts, and the new process opens up to 60.
Worst case per instance during overlap: 20 (old, draining) + 60 (new) = 80. With one instance per region being replaced at a time, that is 2 instances at 80 and 22 instances at 20, giving 22 x 20 + 2 x 80 = 440 + 160 = 600. That exactly equals the database limit, leaving zero headroom. Any additional connection, such as an admin session, a monitoring probe, or a second instance overlapping because health checks run faster than 90 seconds, pushes past 600 and the database refuses new connections. Refused connections surface as errors across all instances, not just the ones being replaced, which matches the previous 6-minute outage: the failure is global and persists until the rollout finishes.
The risk is structural, not bad luck. The math lands on the ceiling with no margin, and the simultaneous two-region deploy doubles the overlap count for no benefit.
2. Propose a rollout plan that keeps the change safe, and state the guardrails you would add.
Read the worked answer
Several changes, each reducing a specific risk:
- Stagger regions. Deploy region A fully, verify, then region B. This halves the overlap count and means a failure affects one region.
- Add a canary. Replace one instance, hold for at least 5 minutes, check error rate and connection count, then proceed. This catches the problem before it is fleet-wide.
- Lower the target. A pool of 60 per instance is not justified by the stated load; if it is, the total budget must be recalculated. With 24 instances, 600 / 24 = 25 connections per instance if all instances run at once, so 60 per instance is only safe if overlap is tightly controlled. Consider a smaller increase, e.g. 20 to 30, giving 24 x 30 = 720 which still exceeds 600, so the real answer may be to add a connection pooler rather than raise per-instance limits.
- Add a proxy or pooler in front of the database so instance count and connection count are decoupled.
Guardrails: a pre-deploy check that computes worst-case connections and fails the pipeline if it exceeds 80% of the database limit; an alert on active connections above 500; a deploy pause if error rate exceeds baseline; and a documented rollback that restores the old config without a second rollout.