RTO 30 minutes on a 40% DR budget, with region-scoped encryption keys
A payment-adjacent SaaS provider runs entirely in region A: 12 application ECS instances behind an ELB, four RDS for MySQL instances, 30 TB of encrypted EVS volumes and 5 TB of receipt PDFs in OBS. The regulator requires RTO 30 minutes and RPO 5 minutes for the application, and RPO 15 minutes for receipts; a full-region failure must not lose data. All volumes are encrypted with customer-managed keys created in region A. The CTO's plan is "nightly CBR backups replicated to a second region". Budget allows roughly 40% of production run-rate, so active-active across two regions is not affordable.
1. Evaluate the CTO's plan against the RTO and RPO targets, and propose a tiered alternative.
Read the worked answer
The plan fails both targets, and buying more backups will not fix it.
Nightly backups give an RPO of up to 24 hours against a 5-minute requirement, and restoring 30 TB of EVS from a vault plus rebuilding 12 instances and four databases cannot complete in 30 minutes - restore throughput and instance boot time dominate. CBR is the right tool for logical corruption and ransomware, and the wrong tool for regional loss.
Tier the recovery by workload:
- Application tier → warm standby. Keep a minimum-scale scaling group (three instances) in region B from the same golden image, an ELB already configured, and DNS pointing at region A. On failover, scale out and cut DNS. This is minutes, and the standing cost is a small fraction of production.
- Database → continuous replication. Use DRS to keep a region B RDS current. The real RPO is the replication delay, so measure it and alert on it; a nominal "real-time" replication that is 30 minutes behind under load meets no target.
- Block storage → SDRS cross-region replication for the 30 TB, or better, make the application tier stateless and move bulk content to OBS, which removes 30 TB from the recovery path entirely.
- Receipts → OBS cross-region replication, which gives minute-level RPO at object-storage prices.
Keep CBR in place as well - it addresses a different failure mode than region loss, and auditors will ask about both.
2. The security team asks how the DR region will handle the encryption keys. What are the traps?
Read the worked answer
The trap is discovering the key problem during a failover.
CMKs are region-scoped: a key created in region A cannot be used to decrypt in region B. So an encrypted EVS snapshot, backup or database copy replicated into region B is only as recoverable as the key material you have there.
Plan explicitly:
- Create the required CMKs in region B before they are needed, and decide the key-material strategy in advance. Importing identical key material lets existing ciphertext be read in region B; otherwise re-encryption under a region B key must be part of the pipeline, not an afterthought.
- Confirm that every replication and restore job - SDRS, DRS, CBR vault policy, OBS cross-region replication - references a key that exists in the target region, and that the service agency used by those jobs still has the grant to use it.
- IAM identities are account-level, but permissions are attached per project/region. Verify the region B project has the roles, the agency trusts and the policy statements the failover automation needs; a script that assumes a role successfully in region A may be unauthorised in region B.
- Rotate any access keys used by DR automation, and do not let the whole runbook depend on one hard-coded credential held by one engineer.
Then prove it: restore one encrypted volume and one encrypted database backup in region B, and record the evidence. An untested key path is the single most common reason a well-designed DR plan fails on the day.
3. How do you validate the 30-minute RTO and 5-minute RPO before the auditor does, and where does DNS fit?
Read the worked answer
The only credible evidence is a timed, unannounced drill - and DNS is usually the thing that blows the budget.
- Measure, do not assume. Time the failover end to end: decision, DNS change, instance scale-out, database promotion, application health checks. Most breaches come from the human steps and the database promotion, not from instance boot. Repeat until two consecutive drills land under 30 minutes with margin.
- RPO is a replication metric. Read the actual DRS delay and OBS replication lag at the moment of the drill, and record the peak under production write load. Test at peak, not at 03:00, because that is when the failover will happen.
- DNS TTL is an RTO input. Short TTLs reduce how long clients cache the old endpoint, but recursive resolvers may ignore very low values. Use health-checked GSLB so the cutover is automatic, and treat the TTL window as a known part of the 30 minutes rather than a surprise.
- Test the failure you did not design for: a partial failure where region A is degraded but not gone. The failover runbook must include a deliberate decision point with a named authority, and a split-brain check before the region B database accepts writes.
Finally, reconcile the drill result against the 40% budget: if the warm standby comes in well under the cap, spend the remainder on the database replication link and a second drill per year, not on redundant idle instances.