Async Replication RPO on a Saturated MPLS Link
A regional distributor runs SAP ERP on a 30 TB Oracle database. RPO is 15 minutes and RTO is 4 hours. Primary site is Dallas; DR site is 900 km away in Atlanta. The existing 200 Mbps MPLS link between sites is 55% utilized at peak. Daily change rate is 1.8 TB. Nightly full backup to tape completes in 11.5 hours inside a 12-hour window; a 2 TB tablespace restore last tested 14 months ago took 9 hours. The DR site has an older array with 40 TB usable and eight spare hosts, but no automated failover. Management wants to cancel tape and rely on array-based async replication plus snapshots. The network team says replication will fit because 1.8 TB/day is only about 167 Mbps.
1. Check the network team's math and decide whether async replication alone can meet the 15-minute RPO and 4-hour RTO.
Read the worked answer
The network team computed the daily average, not the peak or the RPO interval. 1.8 TB/day is 1.8 x 10^12 x 8 / 86,400 = 166.7 Mbps average. The link is 200 Mbps and already 55% utilized at peak, leaving about 90 Mbps of headroom. Replication must also handle the change rate during busy periods, not a 24-hour average, and it must carry the initial 30 TB copy.
At 90 Mbps, 30 TB takes 30 x 10^12 x 8 / 90 x 10^6 = 2.67 million seconds, about 31 days. Even at the full 200 Mbps it takes about 14 days, so seeding is mandatory. For RPO, 15 minutes of change at 1.8 TB/day is 18.75 GB; at 90 Mbps that takes about 28 minutes, which misses the 15-minute RPO before any burst. At 166.7 Mbps it just fits 15 minutes with zero headroom.
Conclusion: async replication alone over the current link cannot guarantee a 15-minute RPO. Options: upgrade to at least 500 Mbps, enable compression/dedup on the replication stream, replicate only changed blocks with snapshots, and seed the DR array by physical transport. RTO 4 hours also requires automated failover and validated runbooks, which the old DR array does not currently provide.
2. Design a backup and DR architecture that meets the objectives and removes tape dependency without creating a single point of failure.
Read the worked answer
Use a layered design, not a single replication method:
- Production: snapshots every 15 minutes with 24-hour retention for logical corruption and accidental deletion.
- DR: array-based async replication of the 30 TB database to the Atlanta array with a documented lag budget; add log shipping or database-native replication for transaction consistency.
- Backup: daily incremental-forever backups to disk at the DR site, plus a weekly immutable copy to object storage or tape kept offline. Array replication alone is not backup because it replicates deletions and corruption.
- Restore: keep the tape system until at least two successful quarterly restore tests and one full DR failover are complete; do not cancel tape on a promise.
- Network: reserve QoS for replication, compress/dedupe, and seed initially.
This meets 3-2-1: production data, DR copy, offline/immutable copy; two media types; one offsite. The 15-minute RPO depends on link capacity and snapshot/log frequency, so measure actual lag continuously.
3. How would you test and validate the DR plan, and what metrics would you report?
Read the worked answer
Test in three levels. First, quarterly restore tests of a 2 TB tablespace into an isolated DR host, measuring elapsed time and comparing to the 4-hour RTO. Second, a semi-annual non-disruptive DR test: present cloned replicated volumes to DR hosts, start the application against a test database, run validation scripts, and measure RPO lag at the moment of the simulated disaster. Third, an annual full failover with business users, including DNS/load-balancer cutover and fallback.
Report: actual RPO lag (max and 95th percentile), actual RTO by tier, restore success rate, number of runbook deviations, and aging of backups. Track the 9-hour tablespace restore as the baseline and require improvement. A DR plan that has not been restored is a hypothesis, not a control.