DR Strategy for Tiered Workloads with a Fixed Budget
A financial services firm runs 220 VMs across three tiers. Tier 1 has 30 VMs for payment processing with an RTO of 15 minutes and RPO of 5 minutes. Tier 2 has 70 VMs for reporting with RTO 4 hours and RPO 1 hour. Tier 3 has 120 VMs for batch jobs with RTO 24 hours and RPO 24 hours. The current DR plan is tape backup to an offsite vault; the last annual restore test took 38 hours for tier 1. Regulators require DR testing twice per year with documented results. The firm has approved an 18% year-over-year increase in DR budget, but not enough for active-active across all tiers. Leadership asks whether to use pilot light, warm standby, or multi-site active-active per tier. The infrastructure team can automate provisioning with infrastructure as code, but failover runbooks are manual and have not been tested in 14 months. Data replication currently uses asynchronous log shipping with no automatic lag monitoring.
1. Map each tier to a DR pattern and justify the choice against RTO, RPO, and the budget constraint. Include the tier-1 tape-restore gap.
Read the worked answer
Tier 1 should use warm standby, not tape and not full active-active. A pre-provisioned standby environment with continuous asynchronous log shipping can meet an RPO of 5 minutes if lag is monitored and bounded; semi-synchronous replication may be needed if the RPO is strict. RTO of 15 minutes requires automated DNS/load-balancer cutover, pre-created compute, and tested runbooks. Tape restore at 38 hours is 152 times the RTO and is not a viable tier-1 strategy.
Tier 2 should use pilot light: database replica, network, identity, and core infrastructure are always present, while application servers are provisioned by infrastructure as code during failover. That supports a 4-hour RTO and 1-hour RPO at much lower cost than warm standby for 70 VMs.
Tier 3 should use backup and restore, accepting a 24-hour RTO and RPO. Active-active for all tiers is not justified by the 18% budget increase and would add data-consistency complexity. The tier-1 warm standby plus tier-2 pilot light consumes most of the increase, but it closes the regulatory and customer-impact gap where it matters.
2. Identify the most significant residual risk in your recommended design and the controls you would add before the next regulatory test.
Read the worked answer
The largest residual risk is not infrastructure cost; it is an untested failover that either fails silently or creates split-brain. The runbooks are manual and have not been exercised in 14 months, while replication has no lag alerting. A warm standby that has never been failed over is a hypothesis, not a control.
Add a quorum witness and fencing so the standby cannot accept writes while the primary is still active. Automate health checks and replication-lag alerts, and require a documented go/no-go decision for tier 1. Lower DNS TTLs before the test, validate dependency maps, and perform a full failover and failback game day at least twice before the regulatory test. Include data-integrity checks after cutover, not just VM availability. Capture timestamps for detection, decision, cutover, and validation so the achieved RTO and RPO can be proven. Keep a manual abort path and a rollback runbook. These controls convert the design from a paper plan into an auditable capability.
3. Define three quantitative readiness measures to report to the board after each test.
Read the worked answer
Report three measures. First, achieved RTO and RPO per tier versus target, with the detection, decision, and cutover times broken out so delays are visible. Second, percentage of failover steps that are automated versus manual, plus the number of critical dependencies that were untested or discovered during the exercise. Third, restore and data-integrity verification success rate, including the number of records or transactions reconciled after failover.
Support these with a simple pass/fail against the 15-minute, 4-hour, and 24-hour targets and a trend line across the two required annual tests. The board should see whether the 38-hour tape gap is closed, whether replication lag stayed inside the RPO, and whether remediation actions from the prior test were completed. Avoid reporting only "DR test completed"; that hides whether the capability actually met its targets.