A 15-Minute RPO Over a Contended 500 Mbps Circuit
Vantage Retail's primary site (Site A) runs a 6-node cluster hosting 320 VMs with 40 TB consumed. A secondary site (Site B) 60 km away has a 4-node cluster with 32 TB of usable free capacity and 40 VMs of its own. The sites are linked by a 500 Mbps MPLS circuit with 4 ms round-trip latency, but measured throughput falls to about 220 Mbps between 08:00 and 19:00. Tier 1 is 120 VMs / 18 TB with a 15-minute RPO and a 2-hour RTO; tier 2 is 200 VMs / 22 TB with a 24-hour RPO. Tier 1's daily change rate is about 1.9 TB. Today the team takes local Protection Domain snapshots every four hours and has never replicated anything offsite. Site B has no compute headroom beyond N+1 for its own 40 VMs, and uses a different IP addressing scheme from Site A.
1. Is a 15-minute asynchronous RPO achievable for tier 1 over this link? Show the math and the design changes you would make.
Read the worked answer
The steady state is borderline; the initial copy and the business-hours dip are what break it.
- Change rate: 1.9 TB/day ≈ 79 GB/hour ≈ 20 GB per 15-minute interval.
- At the degraded 220 Mbps figure (~27.5 MB/s), 20 GB takes ≈ 730 s ≈ 12 minutes of the 15-minute window - leaving ~3 minutes for retransmits, metadata and the Next snapshot. That is not a design margin.
- At the nominal 500 Mbps (~62 MB/s) it is ~5.4 minutes, which is workable but still shares the circuit with production traffic unless QoS reserves bandwidth.
- Initial seeding over the wire is the killer: 18 TB at 220 Mbps ≈ 7.5 days; at 500 Mbps ≈ 3.3 days. Seed the first copy out of band (seed node/device or a controlled offline copy), then enable incremental replication.
Design changes: reserve committed bandwidth/QoS for replication on the MPLS circuit; keep the 15-minute schedule only for tier 1 and a 24-hour schedule for tier 2; consider NearSync (1-15 minute RPO) only if the circuit is upgraded and its shorter retention is acceptable. Metro Availability is the wrong tool here - although 4 ms RTT is inside the synchronous latency budget, the circuit is shared and contended, and Site B cannot host tier 1 at all.
2. Can Site B meet the 2-hour RTO for tier 1? What would you tell the business?
Read the worked answer
Not today, for two independent reasons - capacity and orchestration.
Capacity: Site B has 4 nodes already running 40 VMs at N+1, with only 32 TB of usable storage free. The 18 TB of tier-1 data fits in storage (assuming RF2 on the DR side), but there is no CPU or memory to power on 120 additional VMs. A 2-hour RTO requires the compute to be resident and idle, not ordered after the disaster. The honest answer is either add nodes to Site B for a warm standby of tier 1, or accept a cold-site RTO measured in days for the remainder.
Orchestration: even with capacity, a 120-VM power-on storm will saturate host boot I/O and storage. Recovery must be runbook-driven (Leap or scripted), sequenced by dependency - database, then app, then web - with boot delays, and rehearsed.
Re-IP and DNS: Site B uses a different address scheme, so every tier-1 VM needs a re-IP/DNS plan and matching application config; test that plan with a real failover of a small subset, not a tabletop exercise. Lastly, tell the business the RPO is measured for replication lag, not for application consistency: a 15-minute-old crash-consistent copy of the database may need recovery steps at the DR side.
3. Design the protection plan across both sites and define how you would prove it works.
Read the worked answer
Plan:
- Local first: protection domains on Site A with frequent snapshots (hourly for tier 1, every 4-6 hours for tier 2) so routine restores never touch the WAN.
- Remote second: asynchronous replication of tier 1 to Site B on a 15-minute schedule once seeded, tier 2 on a 24-hour schedule, both with retention sized on Site B before go-live - 18 TB replicated plus retained snapshots must sit inside the 32 TB usable.
- Bandwidth: QoS for replication, monitored lag as an alerting metric; if lag exceeds one interval for more than an hour, the RPO is already breached and someone must know.
- Recovery: documented runbook with VM boot order, IP/DNS remapping, and a defined decision-maker for declaring disaster.
Proof:
- Seed, then confirm steady-state replication lag stays under 15 minutes during business hours for a full week, including month-end.
- Perform a real failover of a representative 10-VM slice of tier 1 to Site B, measure time-to-service against the 2-hour RTO, and fail back.
- Test a single-VM restore from a local snapshot to prove the small-recovery path, since most real recoveries are single-VM, not site-wide.