Month-End Replication Lag and Snapshot Growth
Northline Retail operates 620 stores and runs its order-management platform on a Dell PowerStore 5000T at headquarters, with asynchronous replication to a PowerStore 3000T at a DR site 480 km away, and NetWorker backups to a PowerProtect DD3300. The WAN link is 1 Gbps and carries replication, backup catalog traffic, and store polling. Finance requires a 15-minute RPO for 180 volumes and a four-hour RTO. During the three-day month-end close, replication lag rises from under two minutes to almost three hours, the source array reports 78% snapshot space consumption, and backup jobs miss their 12-hour window by four hours. Some hourly snapshots are six days old despite a 48-hour retention rule. The network team sees the WAN link at 92% utilization from 22:00 to 03:00. No budget exists for a second WAN circuit this quarter.
1. What evidence would you collect first, and which two causes are most likely? Explain how the symptoms distinguish them.
Read the worked answer
Collect four evidence sets before changing anything: PowerStore replication metrics per volume or volume group (lag, throughput, change rate, snapshot space), WAN and QoS counters on the 1 Gbps path, NetWorker job throughput and start times, and the actual protection-policy assignment and age of every snapshot.
The two most likely causes are WAN contention and a snapshot-policy/space problem. Contention fits because lag climbs only during month-end, when batch change rate rises while the link is already at 92%. A 1 Gbps link provides roughly 100–125 MB/s before overhead; if batch writes exceed that, async replication queues regardless of array health. The backlog then takes hours to drain even after the batch ends.
The snapshot symptom is separate. Six-day-old hourly snapshots mean the 48-hour rule is not being enforced on every volume: a volume may sit in a different protection policy, a manual snapshot may be retained, or expiry may be blocked by a dependent operation. The backup overrun points to shared-path contention, not necessarily DD3300 capacity. Distinguish by correlating lag with WAN utilization and backup start time; if lag persists after backup stops, investigate replication change rate and array latency. Preserve the evidence so the cause is proven, not guessed.
2. Propose a remediation plan that can be completed without a new WAN circuit. Prioritize actions and state the expected effect on RPO, backup window, and snapshot space.
Read the worked answer
Priority 1 is to stop competition on the shared path. Move the NetWorker catalog and any backup traffic off the replication WAN, or apply QoS so replication receives a guaranteed class. Stagger backup start times and throttle backup throughput so it leaves headroom for replication. If the RPO can tolerate a documented month-end exception, schedule the heaviest batch replication after the backup completes rather than letting lag silently reach three hours.
Priority 2 is to fix snapshots. Audit every volume's protection policy, delete orphan or manual snapshots after approval, align hourly retention to 48 hours, and alert at 60% snapshot-space use. This is a policy correction, not a capacity purchase.
Priority 3 is to reduce replication volume. Enable array-based compression or deduplication if licensed and validated, exclude temp or scratch volumes, and stagger replication schedules by volume group instead of a single 15-minute tick for all 180 volumes.
Expected effects: the backup window should return under 12 hours once contention is removed; snapshot space should fall after policy correction; replication lag should return to minutes outside batch peaks. Measure after each step and do not bundle all changes into one outage, because then you cannot tell which action worked.
3. The CIO asks whether to move to synchronous replication for all 180 volumes. Give a recommendation with conditions.
Read the worked answer
Recommend no for all 180 volumes. Synchronous replication makes each write commit depend on the remote array. A 480 km path adds roughly 5 ms round-trip latency before array and switch overhead, which would likely violate the 1.2 ms write SLO for order management and couple headquarters availability to the DR site. It also requires sustained bandwidth for peak writes; if the month-end change rate exceeds the 1 Gbps link, synchronous writes will queue or fail, which is worse than asynchronous lag.
Conditions under which to reconsider: a dedicated low-latency fiber path rather than the shared WAN, measured round-trip latency under about 5 ms, peak write throughput below 50% of link capacity, and explicit business acceptance of added application latency. Use synchronous replication only for a small tier-0 subset on that dedicated path.
A better design is asynchronous replication at 15 minutes for the 180 volumes, application-consistent snapshots, and a tested four-hour DR runbook. If the CIO wants stronger protection, invest first in DR automation and validation, not in stretching synchronous replication across a shared 1 Gbps WAN.