Active-Active Alletra 9000 Design with Peer Persistence
NorthBridge Capital runs 2,200 VMware VMs on three HPE Synergy frames across two data centers 38 km apart. The primary site uses an HPE Alletra 9000 array; a second Alletra 9000 sits in the DR site. Dark fiber provides 2.1 ms RTT and 16 x 16 Gb FC links. The business requires active-active access, RPO 0, RTO 15 minutes, and local write latency below 1 ms; cross-site writes must remain under 5 ms. During a recent test, a controller failure at the primary site caused hosts to lose paths for 95 seconds and a subset of VMs restarted on the DR side. The team has proposed putting the Peer Persistence quorum witness VM in the primary site to reduce management traffic, and using FCIP over the existing WAN for replication because dark fiber ports are scarce. A change freeze starts in three weeks.
1. Where should the Peer Persistence quorum witness be placed, and what happens during a primary-site failure if it is placed as proposed?
Read the worked answer
Place the quorum witness at an independent third site with separate power and network paths. The witness is an arbitrator, not a data path: it lets the surviving array and the hosts determine whether the peer array is truly unavailable or the site is merely isolated. If the witness runs in the primary site, a primary failure takes the witness with it. The DR array then has no majority to authorize automatic promotion, so Peer Persistence may not fail over automatically, or worse, both sides could attempt to act independently during a network partition. Management traffic reduction is not a valid reason to weaken quorum. The witness VM should run on a small management cluster that is not part of the protected VMware cluster, with low-latency connectivity to both arrays. During a real primary failure, hosts that have paths to both arrays should see the DR array transition to active and continue I/O without a VM restart. The 95-second path loss suggests the host layer was not correctly zoned or configured for both arrays, so fixing quorum alone will not meet the 15-minute RTO.
2. The team wants FCIP because dark fiber ports are scarce. Evaluate the replication transport against the 5 ms cross-site write budget.
Read the worked answer
Synchronous Peer Persistence acknowledges a host write only after both arrays have it, so every write pays the round-trip plus any transport buffering. A 2.1 ms RTT leaves about 2.9 ms of margin against the 5 ms budget. Direct dark fiber FC is preferable because it adds minimal latency and jitter. FCIP encapsulates FC frames over IP, adds gateway processing and buffering, and can introduce variable latency under congestion; the 5 ms target becomes fragile. Also check bandwidth: 250,000 IOPS at 8 KB with a 70/30 read/write mix is roughly 75,000 write IOPS, or about 600 MB/s of replicated write traffic before peaks. A 10 GbE FCIP link cannot carry a 16 Gb FC line rate, so use 25/40/100 GbE with QoS, jumbo frames, and dedicated links. If FCIP is unavoidable, validate with real workloads, set a latency ceiling, and monitor buffer credits and drops. Otherwise retain dark fiber and add FC ports rather than compromise the RPO 0 design.
3. Why did hosts lose paths for 95 seconds, and how should the host and SAN layer be configured to prevent VM restarts?
Read the worked answer
The symptom points to hosts not having a complete, correctly claimed path set to both arrays. Peer Persistence depends on hosts seeing both arrays through FC zoning and on ALUA reporting the active/optimized path. If hosts are zoned only to the primary array, or if the multipath policy does not support ALUA transitions, the DR array cannot take over transparently. Configure VMware NMP with the ALUA SATP and Round Robin PSP, verify claim rules, and confirm each host has paths to both arrays in the same fabric zones. Present the same LUNs from both arrays with matching LUN IDs and correct host persona. Test a controlled controller failure and confirm path states change without All-Paths-Down. Also review vSphere HA admission control: if datastores became unavailable, HA may have restarted VMs unnecessarily. The fix is not a longer timeout; it is correct dual-array zoning, ALUA multipathing, and failover testing before the change freeze.