Active-Active DR for Core Banking with a Third-Site Quorum
A bank runs its core ledger on a 30 TB OceanStor Dorado at DC1. A second Dorado at DC2, 45 km away, is connected by dark fiber with measured 2.8 ms RTT and 0.4 ms jitter. The ledger requires RPO 0 and RTO 15 minutes. A 200 TB document store on the same arrays can tolerate RPO 1 hour and RTO 4 hours. A third site, DC3, is 80 km away and can host a quorum server but no production storage. The bank also has a backup target at DC3. The network team can guarantee the fiber latency but not a second fiber path; fiber cuts have happened twice in five years. The application team wants VMware HA to restart ledger VMs at DC2 after a site failure. The storage team must present a design that avoids split-brain and meets the RPO/RTO.
1. Select the replication technology for the ledger and the document store. Justify each choice against the RPO/RTO and link constraints.
Read the worked answer
For the ledger, use HyperMetro active-active between DC1 and DC2. The requirement is RPO 0 and RTO 15 minutes; asynchronous replication cannot meet RPO 0, and a restart-based approach would struggle with 15 minutes. The 2.8 ms RTT and 0.4 ms jitter are within a synchronous active-active design's latency budget, but the design must account for write latency increasing on the round trip and for cache and queue behavior. HyperMetro also lets both sites serve I/O, so a site failure does not require a restore.
For the document store, use HyperReplication asynchronous replication. It can tolerate RPO 1 hour and RTO 4 hours, avoids imposing synchronous latency on 200 TB of less-critical data, and is more economical over the single fiber. Do not put the document store in the same HyperMetro pair as the ledger; it would expand the failure and latency domain.
Both tiers still need independent backup at DC3. If the fiber latency were higher or jitter unstable, I would challenge the ledger design rather than weaken RPO. The single fiber path is a risk, so the design must state what happens during a fiber cut and how operations continue.
2. Where should the quorum server sit, and how should the design handle split-brain and VMware HA?
Read the worked answer
Place the quorum server at DC3, not at DC1 or DC2. With a third site, a link failure between DC1 and DC2 does not let both sites believe they are authoritative; the quorum site breaks the tie. DC3 also hosts backup, but keep quorum and backup on separate roles and credentials where possible. Define a preferred site for ledger access and application failover, and document what happens if DC3 is lost while DC1–DC2 is healthy, and if both the inter-site link and DC3 are lost.
VMware HA across sites is not a substitute for array quorum. Restarting VMs at DC2 while DC1 is still writing can cause dual access. Use application-level or site-aware clustering supported by the vendor; if vMSC is used, validate the array, ESXi, and application support matrix. Test link failure, quorum failure, array failure, and controlled site switchover before go-live.
RTO 15 minutes must include DNS, application, and database validation, not just storage. A design that meets storage RPO but needs 40 minutes of manual database recovery does not meet the business requirement.
3. Outline the DR test plan and the evidence you would present to the business.
Read the worked answer
Test in layers, non-disruptively first: path failure, single controller failure, and replication lag monitoring. Then a planned switchover with application owners: stop writes, confirm HyperMetro status, switch preferred site, restart or validate the ledger, and measure RTO. Test unplanned site failure only in a window with rollback: isolate DC1, confirm quorum behavior, confirm DC2 takes writes, and measure data loss, which should be zero for HyperMetro. For the document store, measure asynchronous lag and restart time against RPO 1 hour and RTO 4 hours.
Test DC3 quorum loss and inter-site link loss separately. Evidence should include timestamps for last write, failover start and end, application transaction checks, path state, replication lag, and any manual steps. Include a restore test from the DC3 backup, not just replication. Record defects and re-test.
The business should sign off on the actual RTO including application and network steps, not the storage array alone. A quorum or fiber failure that requires manual intervention must appear in the runbook with an owner and a decision deadline.