Thirty-Minute RPO on a Shared 250 Mbps WAN
A primary site protects 60 TB of data. The WAN link is 250 Mbps and carries both replication and user traffic. Daily change rate averages 1.2 TB, but month-end batches can produce 4 TB in a 6-hour window. Current protection is snapshots every 6 hours retained 7 days, and asynchronous replication every 4 hours. Tier-1 applications (20 TB) require RPO 30 minutes and RTO 4 hours. During a DR test, failover took 9 hours: 400 VMs were powered on sequentially, DNS was updated manually, and the runbook was outdated. Replication lag alerts occur daily between 01:00 and 03:00 and the backlog reaches 3.5 TB. WAN utilization peaks at 95%.
1. Is the current RPO met for tier-1? Show the bandwidth math for 30-minute replication and explain the lag.
Read the worked answer
The current 4-hour replication interval cannot meet a 30-minute RPO. Even if replication completes, the maximum data loss window is up to 4 hours, not 30 minutes.
For a 30-minute RPO, tier-1 changes must be shipped every 30 minutes. At an average 1.2 TB/day across all data, the average rate is 1.2 TB x 8 x 1024 Gb / 86,400 s = about 114 Mbps. If all of that change belonged to tier-1, a 30-minute interval would send about 25 GB, or 200 Gb, in 1,800 s, which is about 111 Mbps. Nominal 250 Mbps looks sufficient on average, but the link is shared and peaks at 95%. Month-end changes of 4 TB in 6 hours require 4 TB x 8 x 1024 Gb / 21,600 s = about 1.52 Gbps, far above the link. That is why the backlog reaches 3.5 TB and lag alerts appear during the nightly window. Average bandwidth is not the design constraint; burst headroom and QoS are.
2. Recommend changes to snapshot/replication schedules, retention, and network policy to meet the RPO.
Read the worked answer
Move tier-1 to 30-minute snapshots and 30-minute replication, but only after reserving bandwidth. Put replication traffic in a dedicated QoS class with a guaranteed rate and a ceiling so user traffic cannot starve it. Stagger snapshot start times across containers so 40 VMs do not hit storage simultaneously. Keep longer retention at lower frequency: 30-minute snapshots for 24-48 hours, hourly for 7 days, daily for 30 days, and monthly for compliance. Do not retain every 30-minute snapshot for 7 days; that multiplies metadata and storage cost.
For the month-end burst, either increase WAN capacity, use compression/deduplication on replication, or pre-seed and schedule the batch window so replication has priority. If 4 TB in 6 hours must be protected, the link cannot do it at 250 Mbps; the business must accept a longer RPO for that batch or fund more bandwidth. Validate the new RPO by measuring replication lag, not by checking that a job was scheduled.
3. What changes to the DR runbook and test plan would address the 9-hour RTO failure?
Read the worked answer
The 9-hour failover is an orchestration problem, not a replication problem. Powering on 400 VMs sequentially and updating DNS by hand cannot meet a 4-hour RTO. Build a tiered runbook: power on tier-1 first in parallel waves, then tier-2, with dependency order documented. Automate DNS changes, IP remapping, and validation checks. Pre-stage the DR site with sufficient capacity and pre-configured networking so recovery does not wait on manual provisioning.
Measure the test against the RTO: time to declare disaster, time to make data available, time to power on each wave, time to DNS cutover, and time to application validation. Record actual timings and fix the top bottlenecks. Run a partial failover test for tier-1 monthly and a full test quarterly. The key judgement is that RPO is about data shipping frequency and bandwidth, while RTO is about people, automation, and runbook quality; fixing replication alone will not fix a 9-hour manual failover.