Multi-Region Ambition on a Single-Region Budget
A company's leadership wants "multi-region active-active" after a regional outage took their product down for six hours. The product is a read-heavy catalog with a smaller transactional component. Current architecture: one region, a managed relational database, and a stateless web tier. Requirements stated: 99.99% availability, recovery from regional failure within 15 minutes, and no data loss for completed transactions. Engineering estimates full active-active at roughly 3x current cost and 9 months, largely from making the relational database multi-writer. The finance team has approved a 40% budget increase. The company has never run a failover drill. A significant portion of the catalog is static and changes daily; transactions are low-volume but must not be lost.
1. Challenge the "active-active" framing. What availability strategy actually fits these requirements and budget?
Read the worked answer
Active-active is being used as a proxy for "don't go down again," but the requirements don't demand it.
- 99.99% and 15-minute regional recovery with zero transaction loss are achievable with warm standby (active-passive): a second region with the stateless tier running, the database replicated, and a tested failover procedure. That is far cheaper than multi-writer and avoids the hardest problem (multi-writer relational consistency).
- The read-heavy catalog can be served from replicated read capacity and cached/CDN-delivered, since static content changes daily - it doesn't need active-active writes.
- The transactional component is low-volume, so the cost of synchronous-ish replication to a standby region is modest, and zero-loss is attainable with proper replication mode.
Recommendation: pursue warm standby + tested failover, not active-active. It meets the stated SLOs within the 40% budget. Active-active is a later optimization if latency or capacity demands it.
2. The team has never run a failover drill. How does that change your risk assessment and rollout plan?
Read the worked answer
It changes the plan fundamentally: an untested failover is a hypothesis, not a capability.
- The six-hour outage happened despite (presumably) having a single region - the lesson isn't "we lacked a second region," it's "we couldn't recover quickly." A standby that has never been failed over to may not meet 15 minutes.
- Rollout should sequence: (1) build the standby and replication; (2) run a scheduled, announced failover drill measuring actual RTO and data loss; (3) fix what the drill exposes; (4) repeat until the drill reliably hits 15 minutes; (5) only then treat the SLO as credible.
- Include failback in the drill - teams routinely test failover and discover failback is the hard part.
- Add monitoring on replication lag and alarms; zero data loss depends on lag being near zero at failure time, which must be observed, not assumed.
Budget the drills as recurring operational work, not a one-time project milestone.