Metro Availability Link Flap and Duplicate Active VMs
Globex operates two data centers, DC-A and DC-B, 4 ms RTT apart, in a stretched cluster with Metro Availability. The witness VM runs at a third site, DC-C. Six protection domains cover 180 VMs. During scheduled WAN maintenance, the link between DC-A and DC-B flapped for 40 seconds while the witness remained reachable from both sites. After the flap, some VMs in DC-A restarted automatically on DC-B, but a few stayed active on both sides for six minutes. The storage team says the witness did its job. The application team says there was split brain. No data corruption has been confirmed, but the incident has triggered an audit finding.
1. Explain what Metro Availability and the witness are expected to do during a 40-second link flap, and why duplicate active VMs could appear.
Read the worked answer
Metro Availability replicates synchronously between sites, so the RPO is effectively zero while the link is healthy. The witness does not carry VM data; it provides quorum so that only one site is allowed to run the protected workload after a site or link failure. With a 4 ms RTT, the design is within a typical metro latency envelope, but a 40-second flap is long enough to trigger failure handling rather than a transient retry.
The expected behavior depends on exactly what each site could reach. If DC-A lost DC-B but still reached DC-C, and DC-B also reached DC-C, the witness should grant quorum to one side and deny the other. The denied side should stop or fence its VMs before the other side restarts them. Duplicate active VMs appear when that fencing is incomplete or when the two sites reach contradictory conclusions. Common causes: the witness was reachable intermittently rather than continuously; one site's management plane was isolated from the witness while storage paths differed; an administrator manually powered on VMs during the event; or the failover completed on one side while the original VMs had not yet been killed on the other. A six-minute overlap is consistent with a delayed fencing or manual restart, not with normal synchronous replication.
Calling it 'split brain' is imprecise unless both sites were actively writing the same protected datastore. The correct question is whether the witness quorum decision was consistent and whether VM fencing completed before restart.
2. What would you check in Prism, logs, and the network to determine whether the witness behaved correctly?
Read the worked answer
Build a timeline from all three layers before accepting either team's conclusion.
Prism and protection-domain state. Check alerts and events for the exact window: witness connectivity, protection-domain status, sync state, and any automatic failover event. Review the VM power state history for the affected VMs on both sides. Confirm whether the restarts were automatic or manual, and which user or service initiated them.
Witness and quorum logs. Verify the witness saw both sites, the timestamps of quorum grants, and whether it lost a path to either site. A witness that stayed reachable from both sites should show a single grant. If it shows two grants or a lost lease followed by a new grant, that is the defect to pursue.
Network evidence. Pull WAN device logs for the 40-second flap. Check whether the loss was symmetric, whether management and storage traffic took different paths, and whether latency or packet loss preceded the flap. Verify the witness path did not traverse the same WAN link. Confirm RTT stayed within design limits outside the event; a 4 ms average can hide spikes.
VM and application evidence. Check whether both copies were writing, whether the application uses a cluster-aware quorum, and whether any shared IP or database lock conflicted. If both copies wrote, treat it as a data-integrity incident and involve the application owner; if only one wrote, it was a fencing delay.
Document timestamps in a single timeline and reconcile clock sync across sites.
3. Recommend changes to prevent recurrence and satisfy the audit finding.
Read the worked answer
The fix is a combination of design, procedure, and evidence.
- Separate the witness path. Ensure DC-C is reachable independently of the DC-A/DC-B WAN. If the witness shares the same provider or conduit, the quorum design is weaker than documented.
- Validate fencing. Confirm that the denied site stops or fences protected VMs before the surviving site restarts them. Test this deliberately in a controlled failover, not just during an incident.
- Define failover policy per protection domain. Some workloads should not auto-restart. For those, require manual approval and a documented runbook. For auto-restart workloads, make sure the application is cluster-aware or single-writer.
- Change management for WAN maintenance. Any maintenance on the inter-site link should either be outside Metro protection windows or explicitly account for failover behavior. A 40-second flap is not a routine event for a synchronous stretched cluster.
- Monitoring. Alert on witness reachability from both sites, protection-domain sync state, and duplicate VM IP or name detection. Correlate with the WAN monitoring system.
- Audit evidence. Produce a post-incident report with the timeline, quorum logs, VM power history, and corrective actions. State clearly whether any dual writes occurred. If they did, include data reconciliation results.
- RPO/RTO review. Confirm that the business owners accept automatic failover for these six protection domains, and document any domains moved to manual or async DR.