Multi-Region Failover Where the Database Is the Real Blocker
A SaaS runs stateless web tiers in us-east-1 and eu-west-1 behind Route 53 failover routing with health checks. The primary database is Aurora MySQL with a cross-region read replica in eu-west-1. RTO target is 15 minutes; RPO is 5 minutes. During a regional test, the team flips DNS to eu-west-1. The web tier comes up healthy, but the app throws errors because it is still pointed at the us-east-1 writer endpoint. Promoting the replica takes 9 minutes. After promotion, writes resume but roughly 12 minutes of recent transactions are missing, and some clients cached the old DNS TTL of 300s. The postmortem notes the failover was "mostly successful."
1. Evaluate whether this failover met the stated RTO and RPO, and identify the gaps.
Read the worked answer
It did not meet either target once you measure end-to-end user impact, not just DNS flip time.
- RTO: DNS flip + health check + 9-minute promotion + app reconfiguration + client TTL expiry (up to 300s) exceeds 15 minutes. The web tier being healthy is irrelevant if the app can't reach a writable database. RTO must be measured from incident start to successful writes, which includes DB promotion and connection-string cutover.
- RPO: 12 minutes of lost transactions vs a 5-minute target. Aurora cross-region replication lag is the cause; if lag exceeded 5 minutes at failure time, RPO was already breached before the failover began.
Gaps:
- App config not part of the failover runbook - the writer endpoint must switch (via Route 53 CNAME, a parameter store value, or a global database endpoint), and this wasn't automated.
- No replication-lag monitoring against RPO - the team had no alarm that lag was eating the RPO budget.
- Client TTL not accounted for in RTO.
"Mostly successful" is the wrong verdict; it's a failed DR test that surfaced exactly the gaps tests exist to find.
2. What concrete changes would you make so the next test plausibly hits 15-minute RTO and 5-minute RPO?
Read the worked answer
- Automate the whole cutover, not just DNS. A runbook automation (Step Functions/SSM document) that: promotes the replica, updates the writer endpoint in Parameter Store/Secrets Manager, signals the app to reload, and only then shifts DNS. Measure each step.
- Use a managed global endpoint if available (Aurora Global Database) so promotion and endpoint redirection are one operation rather than a manual sequence, and the app points at a stable endpoint.
- Monitor replication lag as an RPO alarm. Alarm when lag approaches the 5-minute budget; treat sustained lag as a DR-readiness failure. Consider that RPO is only as good as the lag at the moment of failure.
- Lower DNS TTL and design for fast client re-resolution, or front with a global accelerator that fails over at the network layer with faster convergence than 300s TTL.
- Rehearse with the clock running and assert on the success criteria (first successful write timestamp), not on component health. Re-test until the measured RTO/RPO are consistently inside budget, and fail the test loudly otherwise.