The Backup That Was Never Restored
A financial services firm runs a primary workload in us-east-1 with an RPO of 15 minutes and an RTO of 1 hour. They take nightly EBS snapshots and daily RDS automated backups with a 7-day retention. A compliance audit reveals that the DR plan has never been tested end to end. The RDS instance is a single-AZ db.r6g.2xlarge. Cross-region replication exists only for an S3 bucket holding static assets. During a tabletop exercise, the team estimates that rebuilding the application stack from snapshots and AMIs would take 6 hours, and that the most recent RDS automated backup is up to 24 hours old.
1. Assess the gap against the stated RPO and RTO, and identify the single most dangerous assumption.
Read the worked answer
RPO gap: 15 minutes required vs. up to 24 hours actual for RDS - a 96x miss. Nightly snapshots cannot meet a 15-minute RPO. RTO gap: 1 hour required vs. ~6 hours estimated rebuild - a 6x miss.
The most dangerous assumption is that the backups are restorable and sufficient. Untested backups are a hypothesis, not a control. Specific risks: snapshots may be incomplete or corrupt; IAM/KMS permissions needed to restore may have drifted; the AMIs may reference resources (security groups, secrets, DNS) that no longer exist; and a single-AZ RDS instance means an AZ failure is itself an outage, not just a DR scenario.
The gap is structural, not procedural - no amount of rehearsal makes a 24-hour backup meet a 15-minute RPO. Remediation must change the architecture: enable RDS Multi-AZ (removes single-AZ as a failure mode), turn on automated backups with point-in-time recovery (gets RPO to ~5 minutes), and add cross-region read replicas or Aurora Global Database for regional resilience.
2. Propose a concrete change set to meet RPO/RTO, and describe how you would validate it.
Read the worked answer
Changes:
- Enable RDS automated backups with PITR (RPO ~5 min) and set retention to at least 35 days.
- Convert to Multi-AZ for in-region AZ-failure resilience.
- Add a cross-region read replica (or Aurora Global Database) in us-west-2 to cover regional failure; promote on failover.
- Replace AMI-based rebuilds with infrastructure-as-code (CloudFormation/Terraform) so the stack can be recreated deterministically in minutes.
- Pre-stage networking (VPC, subnets, security groups) and secrets in the DR region so restore does not depend on manual steps.
- Automate the failover runbook and store it in version control.
Validation:
- Run a game day quarterly: promote the replica in the DR region, repoint DNS, and measure actual RTO with a stopwatch.
- Periodically restore an RDS snapshot to a scratch instance and run integrity checks against known row counts.
- Inject failures (kill an AZ, revoke a permission) to confirm the runbook handles drift.
- Record results and feed gaps back into the runbook. An untested plan is not a plan.