HyperMetro resync saturation on a 1 GbE inter-site link
Two data centers 20 km apart run an active-active stretched cluster. Each site has a dual-controller all-flash array with 128 GB cache and 48 SAS SSDs. HyperMetro mirrors 600 VM volumes, 30 TB total, over two 1 GbE links; average production load is 30,000 IOPS with 70% reads and 30% writes, 8 KB average I/O. A quorum server sits in a third site with 25 ms RTT. During fiber maintenance, one inter-site link was lost for 30 minutes. After it returned, the array started resynchronizing 400 GB of dirty data. The remaining link is now 92% utilized, host p99 write latency is 15 ms versus 2 ms baseline, and the monitoring system shows repeated HyperMetro pair 'recovering' events. No disk or controller errors are present. The business requires RPO 0 and RTO 5 minutes.
1. Calculate the production cross-site write bandwidth and the resync time if resync uses 70% of one 1 GbE link. Determine whether the link is the bottleneck.
Read the worked answer
Production cross-site write bandwidth: 30,000 IOPS × 30% writes = 9,000 write IOPS; 9,000 × 8 KB = 72,000 KB/s = 72 MB/s = 576 Mbps. One 1 GbE link carries about 940 Mbps at line rate, so production alone uses about 61%. Resync at 70% of 1 GbE = 700 Mbps = 87.5 MB/s; 400 GB = 400,000 MB; time = 400,000 / 87.5 ≈ 4,571 s ≈ 76 minutes. Combined demand is 576 + 700 = 1,276 Mbps, which exceeds the 940 Mbps link. The link is therefore oversubscribed, and queueing explains the 15 ms p99. The resync is not just background noise; it competes with synchronous mirror writes. Rate-limit resync to leave at least 300–400 Mbps for production, or pause it during peak.
2. Recommend immediate and permanent actions. Address quorum behavior and why adding a third link alone may not fix the 15 ms latency.
Read the worked answer
Immediate: throttle or pause resync, confirm the quorum server is reachable, and check each HyperMetro pair state before any forced failover. If the pair went to a local-write state during the 30-minute link loss and production continued, RPO 0 was already violated; confirm the link-down policy. Permanent: add a second 1 GbE link or upgrade to 10 GbE, but adding bandwidth alone will not fix latency if resync can still consume the new capacity. Configure resync QoS/rate limiting, separate replication traffic if possible, and monitor link utilization, resync progress, and host queue depth. Quorum is at 25 ms RTT; it should not carry data, but verify it is not on the same fiber path. With RTO 5 minutes, test a controlled link failure and confirm the surviving site and quorum behavior.
3. What evidence would confirm that resync, rather than production load, is causing the 15 ms latency?
Read the worked answer
Pause or rate-limit resync and observe p99 latency within a few minutes; if latency falls to near 2 ms while production IOPS stays at 30,000, resync is the cause. Correlate latency with link utilization and resync progress: latency should rise when the link exceeds about 85–90% and resync throughput climbs, not when production alone is 576 Mbps. Check array port queue depth and buffer credits; resync creates large bursts that fill queues. Compare per-pair resync state and host latency by LUN; if all LUNs on the link degrade together, it is the shared link, not one hot LUN. Finally, review the timeline: latency began after the 400 GB resync started, not after a production IOPS increase. If latency stays high with resync paused, investigate array cache or back-end SSDs.