SRDF/A Lag During Month-End with a Shared Test Initial Copy
SRDF/A is configured between two PowerMax 2500 arrays for a 12 TB SG_PAYMENTS storage group. The RDF group uses two 10 GbE RDF ports over a 35 ms RTT FCIP link. During month-end, host writes to the source rise from 90 MB/s to 220 MB/s. Unisphere shows the SRDF/A session is Active but delta sets are growing; cycle time has risen from about 60 s to over 120 s; RDF port utilization is 92%; and the remote copy lag is 18 minutes. The network team says the FCIP link has no errors but is at 85% utilization in the peak hour. The business RPO is 15 minutes. A second RDF group for a test workload shares the same RDF ports and is running a large initial copy. You have a 2-hour window before the next batch.
1. Where is the bottleneck, and what evidence confirms it?
Read the worked answer
The evidence points to the shared RDF transport path, not the array back-end. The production RDF ports are at 92% utilization, the FCIP link is at 85%, and a test RDF group is running a large initial copy over the same ports. The SRDF/A session is still Active, but delta sets are growing and cycle time has stretched from about 60 s to over 120 s, which is what happens when the session cannot drain each delta set within the target cycle. An 18-minute lag against a 15-minute RPO is a direct RPO breach, not merely a performance nuisance.
Confirm with symrdf -g <group> query for session state, consistency, and cycle information; symstat -g <group> -type RDF for port utilization and queue behavior; and Unisphere SRDF topology for the test group's initial copy state. Ask the network team for per-flow or per-port utilization during the peak, not just aggregate link health. If array back-end and cache metrics are normal, the shared RDF ports and FCIP capacity are the constraint.
2. What immediate actions can restore the 15-minute RPO within the 2-hour window, and what are the trade-offs?
Read the worked answer
The fastest lever is the test initial copy. Pause or throttle the test RDF group so production SRDF/A can use the shared RDF ports. That is preferable to changing production SRDF/A parameters because it removes competing load without widening the production RPO.
Sequence:
- Get approval to pause the test initial copy and record its restart point.
- Monitor production lag and RDF port utilization for 15-30 minutes. If lag falls below 15 minutes, keep the test copy paused until after the month-end batch.
- If lag remains above RPO, consider temporarily reducing non-critical source writes or ask the application owner to defer the batch. Do not simply shorten the SRDF/A cycle time; that can increase overhead and make congestion worse.
- If the link remains saturated and the business cannot defer writes, escalate for temporary bandwidth or an additional RDF path. Those are not 30-minute changes, so the test copy pause is the practical first action.
Trade-offs: pausing the test copy delays test readiness; throttling production writes affects the business; changing cycle time can increase RPO and link pressure. Do not claim RPO is restored until consecutive cycles complete within 15 minutes.
3. What permanent design and monitoring changes would you recommend?
Read the worked answer
Separate production and test RDF traffic. Put the test RDF group on dedicated RDF ports or a dedicated link, or schedule initial copies outside month-end and batch peaks. If separation is not possible, use QoS or bandwidth controls so a test initial copy cannot consume production RDF headroom.
For capacity and performance planning:
- Measure peak production write rate and convert it to RDF bandwidth needs, including FCIP overhead and growth.
- Monitor RDF port utilization, FCIP utilization, cycle time, delta-set size, and lag continuously.
- Alert on lag approaching the RPO, not only when the session is inconsistent.
- Document an RPO-protection runbook: pause test copies first, then throttle or defer batch work, then add bandwidth.
- Reassess the two-port design at month-end peak; 92% port utilization leaves little failure headroom, so one port loss could cause an immediate RPO breach.