Failover Test Misses the Order-Processing RTO
Northline Retail runs order processing in us-east-1 with a pilot-light DR site in us-west-2. The business impact analysis sets a 4-hour RTO and 15-minute RPO for order processing. The DR plan relies on asynchronous Aurora replication, a standby auto-scaling group, and nightly snapshots. During a quarterly test on a Tuesday at 14:00 UTC, the team declares a disaster and starts failover. DNS TTL is 3,600 seconds. The runbook says to promote the Aurora replica manually, but the documented console path has changed and the on-call DBA takes 80 minutes to find the CLI command. Replication lag is 38 minutes because the test overlaps a flash sale. After promotion, the application fails health checks until a missing security group rule is added, adding 45 minutes. The team declares success at 20:10 UTC, six hours after the simulated disaster, and reports “DR test passed with minor issues.” The next test is in 12 months.
1. Evaluate whether this test should be considered a pass. Identify the RTO and RPO gaps with arithmetic.
Read the worked answer
Treat the test as a fail, not a pass. The declared RTO is 4 hours; the team took 6 hours 10 minutes from 14:00 to 20:10, missing the objective by 2 hours 10 minutes. The RPO is 15 minutes, but replication lag reached 38 minutes at failover, so up to 38 minutes of orders could be lost - 23 minutes beyond tolerance. The DNS TTL alone can add up to 60 minutes of stale resolution, the manual database promotion consumed 80 minutes, and the missing security-group rule added 45 minutes. Those three items account for 185 minutes of the overrun and are fixable process/configuration defects, not random variance. The test also overlapped a flash sale, which is realistic stress but should have been declared as a peak-load test. Calling it ‘passed with minor issues’ understates a material RTO/RPO breach and would mislead executives and auditors. Record it as a failed test with findings, owners, and a retest date.
2. Recommend the top three changes to meet the objectives under a limited budget.
Read the worked answer
Prioritize the critical path. First, automate Aurora promotion with infrastructure-as-code and a tested CLI/API runbook; this removes the 80-minute manual search and reduces human error. Second, lower DNS TTL to 60 seconds well before failover and use health-check-based routing so clients move without waiting an hour. Third, pre-create and validate security groups, load-balancer rules, and application config in the DR region as part of a golden image, and add an automated post-failover smoke test. Together these could save roughly 185 minutes, bringing 6h10 down to about 3h05, inside the 4-hour RTO. For the RPO, monitor replication lag and throttle noncritical batch traffic during peak; if 15 minutes is mandatory, budget for a globally distributed database or synchronous replication for the order store. Retest during a peak period within 90 days, not 12 months.
3. What should the executive report contain, and what should not be claimed?
Read the worked answer
The executive report should state the actual RTO and RPO achieved, the business impact of the gap, the root causes, and a dated remediation plan with owners. It should include a clear residual-risk statement: until retested, the service cannot be represented as meeting the 4-hour/15-minute objectives. It should not claim success, should not bury the DNS and security-group defects as ‘minor,’ and should not imply that a single successful retest proves readiness. Include a decision request for the replication/RPO funding, a schedule for the next test, and evidence that customer-facing commitments and regulatory notifications were considered. Keep the detailed timeline evidence in an appendix for auditors. The board needs to know that the current plan has a material data-loss exposure and a failover time roughly 50% over target.