Replication Lag on a DD9400-to-DD9400 DR Pair
A DD9400 at the primary site replicates 40 MTrees to a DD9400 at the DR site over a 1 Gbps WAN with 20 ms latency. The initial 20 TB seed was completed two weeks ago using physical media. Daily change is 3 TB, and the source deduplication ratio is 5:1. The RPO is 2 hours. Replication is configured with MTree replication, encryption, and low-bandwidth optimization. After two weeks, replication lag is 9 hours and growing. WAN utilization is only 35%, the target is at 70% capacity, and alerts show replication lag. The customer has not changed backup schedules. They ask whether the WAN is too small, why utilization is low if replication is behind, and whether they should disable encryption or low-bandwidth optimization to catch up.
1. Calculate the expected replication load and determine whether the 1 Gbps WAN is theoretically sufficient for the 2-hour RPO.
Read the worked answer
Daily change is 3 TB. At 5:1 deduplication, the replicated change is about 600 GB per day. 600 GB is 4,800 Gb. At a dedicated 1 Gbps, the transfer time is 4,800 seconds, or 80 minutes, which is within the 2-hour RPO. Even with protocol overhead and encryption, the raw link should be sufficient if replication can use it. The 35% utilization while lag is growing indicates the bottleneck is not simply WAN capacity; something else is limiting replication throughput, such as stream limits, source snapshot/processing time, target ingest, latency effects, encryption overhead, QoS, or errors.
The calculation also assumes all 600 GB must cross the WAN in one window and that the link is available. If backups overlap, if replication is throttled, or if only a few streams are active, the effective throughput can be far below 1 Gbps. The correct conclusion is: the WAN is theoretically adequate, but the replication pipeline is not achieving its potential.
2. List the troubleshooting checks you would perform to explain low WAN utilization with growing lag.
Read the worked answer
Check replication status and configuration first: replication show config, replication show status, and replication show stats to see whether all 40 MTrees are replicating, whether any are in initialize or recovery state, and whether lag is concentrated on a few MTrees. Review replication show performance for throughput and stream usage. Confirm the source is creating snapshots on schedule and that backup jobs are not holding locks or saturating the source disk.
On the network side, check net show stats for interface errors, drops, retransmits, and QoS markings. Verify the 1 Gbps path is actually delivering 1 Gbps and not shaped below 35%. Check latency and loss; 20 ms alone should not prevent throughput, but packet loss can. Confirm encryption and low-bandwidth optimization are functioning and not causing CPU saturation. On the target, check capacity, disk health, ingest throughput, and whether the target is busy with cleaning or other replication sources. Also verify stream limits per MTree and per system; too few streams can leave the WAN idle while each stream waits on latency.
3. Recommend a remediation plan and address whether encryption or low-bandwidth optimization should be disabled.
Read the worked answer
Do not disable encryption to chase throughput unless the security requirement allows it; encryption is usually not the primary cause of 35% utilization. Low-bandwidth optimization is designed for WAN replication and should remain enabled unless testing proves it is CPU-bound on this platform. Instead, address the pipeline: increase replication streams where supported, stagger backup and replication schedules, verify snapshot creation is not delayed, and check QoS or carrier shaping that may be limiting the replication VLAN. If the target is the bottleneck, reduce concurrent replication from other sources or add target capacity.
If lag remains after these checks, consider a controlled reseed of the most lagging MTrees, because a large backlog can amplify latency and slow catch-up. Monitor throughput, lag, and target capacity daily until the lag returns to within the 2-hour RPO. If the WAN is shared and cannot sustain the required rate during backup peaks, add bandwidth or schedule replication outside peak periods. Document that the 1 Gbps link is sufficient in theory, but only if streams, source snapshots, target ingest, and QoS allow replication to use it.