Cross-Region DR for a Database-Backed Application
A financial reporting application runs in Region A with an ApsaraDB for RDS MySQL primary and two read replicas, plus an OSS bucket for generated reports. The business requires an RPO of 5 minutes and an RTO of 30 minutes for a regional failure. The team proposes a standby Region B with an RDS instance restored from backups nightly, an OSS bucket replicated daily, and a DNS switch performed manually by an on-call engineer. A quarterly drill showed the failover took 4 hours.
1. Assess the proposed design against the stated RPO and RTO, identifying each gap.
Read the worked answer
RPO (5 minutes) - not met. Nightly backups give an RPO of up to 24 hours. Daily OSS replication gives a similar worst case for reports. The design misses the target by orders of magnitude.
RTO (30 minutes) - not met. The drill took 4 hours. Contributors likely include: restoring a database from backup (slow and size-dependent), DNS propagation and TTL, manual decision and execution by an on-call engineer, and unverified application configuration in Region B.
Other gaps:
- A manually triggered DNS switch depends on a human noticing the failure and acting, which alone can exceed 30 minutes.
- The standby RDS instance is not kept warm, so application connection strings, schema, and parameter groups may drift.
- No stated plan for the read replicas or for validating data integrity after failover.
In short, the design is a backup strategy, not a disaster-recovery strategy, and does not meet either target.
2. Redesign the data layer to meet a 5-minute RPO and 30-minute RTO. Be specific about replication and promotion.
Read the worked answer
- Use cross-region data replication for RDS so changes stream continuously to Region B rather than arriving nightly. A cross-region read replica or a data-transmission service keeps the standby within seconds to low minutes of the primary, meeting a 5-minute RPO.
- Keep the standby warm. The Region B instance should be running, with the same schema, parameter group, and application configuration, so promotion is a switch rather than a build.
- Automate promotion. Define a runbook that promotes the cross-region replica to primary, repoints the application via a controlled mechanism (a connection endpoint or DNS record with a short TTL), and validates replication lag before cutover. Automate as much as possible so the 30-minute RTO is achievable without a human improvising.
- Replicate OSS continuously or on a short schedule so reports are available in Region B within the RPO.
- Measure replication lag continuously and alert when it approaches the RPO.
3. How would you make the failover decision and DNS switch reliable within 30 minutes, and how would you validate the design?
Read the worked answer
Decision and switch:
- Use health checks plus automated DNS failover, or a documented decision tree with a named owner and a paging path, so detection does not depend on someone noticing.
- Set a short DNS TTL (for example 60 seconds) on the records that must move, and pre-create the Region B records so the switch is a flip, not a creation.
- Keep application configuration region-agnostic (endpoint names, not hard-coded IPs) so no code change is needed.
- Pre-stage credentials, certificates, and security group rules in Region B.
Validation:
- Run drills on a schedule and time each phase (detection, decision, promotion, DNS, verification) against the 30-minute budget.
- Inject failures rather than simulating them, so the runbook is exercised under realistic conditions.
- After each drill, record actual RPO achieved (compare data timestamps) and actual RTO, and fix the slowest phase.
- Track replication lag and DNS TTL behavior as leading indicators between drills.
The 4-hour drill result should be decomposed into phase timings; the largest phase is the first thing to automate.