Regional Failure With a 15-Minute RPO
An order service runs Aurora MySQL in eu-west-1 with one writer and three readers, 7-day automated backup retention, and no cross-region copies. The database is 1.8 TB and grows about 2% per week. The business requires RPO <= 15 minutes and RTO <= 60 minutes for a complete regional failure. The current DR runbook is to restore the latest snapshot in eu-central-1; it was last tested 14 months ago and took 3 h 40 m. The application tier is stateless EC2 deployed by CloudFormation, but the template hardcodes eu-west-1 AMI IDs. DNS is a single Route 53 A record aliased to an ALB in eu-west-1.
1. Which DR strategy actually satisfies RPO 15 minutes and RTO 60 minutes, and what rules out the alternatives?
Read the worked answer
Chosen strategy: Aurora Global Database with a secondary cluster in eu-central-1, plus a warm standby application tier.
Reasoning against the requirements:
- Backup and restore fails RTO. Restoring 1.8 TB took 3 h 40 m last time; even if faster now, it cannot reliably beat 60 minutes.
- Snapshot copy every 15 minutes fails both cost and RTO. Copying 1.8 TB across regions at that cadence is expensive, and the restore time still dominates.
- Aurora Global Database replicates storage asynchronously with typical lag around one second, so RPO is far inside 15 minutes. Promotion of the secondary cluster is a managed operation measured in about a minute for unplanned failover. Accept that asynchronous replication means a small amount of recent writes can be lost; that is within the stated RPO.
- Warm standby, not pilot light. RTO includes application start, DNS TTL, and health-check detection. Keep at least minimal capacity running in eu-central-1 so the app does not need a cold boot after promotion.
Document the actual measured promotion and cutover times, not vendor best-case numbers.
2. What must change outside the database for the 60-minute RTO to be believable?
Read the worked answer
The database is only one dependency. Enumerate everything that is region-scoped:
- AMIs: copy images to eu-central-1 in the pipeline (EC2 Image Builder or
CopyImage), and replace hardcoded IDs with CloudFormation parameters or region mappings, or SSM public parameters per region. - Stack: deploy the application stack in eu-central-1 ahead of time at minimum capacity, or make one-click deployment part of the tested runbook. Nested stacks and cross-stack exports must resolve in the DR region.
- Certificates and DNS: ACM certificates are regional, so issue one in eu-central-1. Use Route 53 failover records with a health check and a TTL of 60 seconds or lower; a long TTL alone can consume the RTO.
- Secrets and config: replicate Secrets Manager secrets to the DR region and point the app at regional endpoints.
- Quotas and dependencies: verify EC2, EIP, and Aurora quotas in eu-central-1, and confirm any third-party allowlists include the DR egress addresses.
Each item is a potential multi-minute surprise on failover day.
3. How would you prove the RTO and keep it true over time?
Read the worked answer
Treat DR as a continuously measured control, not a document:
- Quarterly full game day: promote the secondary, shift DNS, run synthetic order transactions, and record timestamps for detection, promotion, app health, and first successful customer request. Publish the measured RTO.
- Monthly partial test: promote a clone or run a planned switchover during a maintenance window to exercise the runbook without risking production.
- Replication lag alarm: alarm on
AuroraGlobalDBReplicationLagabove a threshold tied to the RPO, not just on cluster health. - Drift checks: run CloudFormation drift detection on the DR stack so an out-of-band console change cannot silently break the standby.
- Automate the runbook in SSM Automation so recovery does not depend on one engineer remembering steps.
If the measured RTO drifts above 45 minutes, treat it as an incident, because the 60-minute target has no margin left for a real event.