Stretched Cluster Across Two Rooms With One SAN
Fabrikam has a two-node Windows Server 2022 failover cluster (NODE01, NODE02) running 60 highly available VMs on CSV storage from a single SAN. The nodes are in separate rooms of the same building, connected by a 1 Gb dedicated heartbeat network and a 10 Gb iSCSI network. Last month a room power failure took NODE01 offline; VMs failed over, but 14 VMs restarted twice because the cluster quarantined NODE02 for 90 seconds during the event. The SAN itself is in the room with NODE01 and lost power for 40 minutes, so all VMs were down despite the surviving node. The business requires RTO of 15 minutes for these VMs and asks whether a third node is needed.
1. Explain why the 14 VMs restarted twice and what cluster setting addresses it, given the described topology.
Read the worked answer
Cause. When NODE01 lost power, the surviving node could not immediately confirm whether NODE01 was truly down or merely unreachable. Cluster heartbeat settings determine how quickly the surviving node declares the peer dead. If the CrossSubnetDelay/SameSubnetDelay and CrossSubnetThreshold are at defaults (1000 ms delay, 5 threshold), detection can take ~5 seconds - but the observed 90-second quarantine suggests the cluster lost quorum or the node was marked Isolated and then rejoined, triggering a second failover cycle: VMs moved to NODE02, then the cluster re-evaluated and restarted them when quorum was re-established.
Fix. With only two nodes, quorum is the real problem. A two-node cluster has no natural majority; when one node is lost the survivor must be sure it owns quorum. Configure a disk witness (if a third independent LUN exists) or, better here, a cloud witness or file share witness on a host outside the affected room. Then set SameSubnetDelay/Threshold (or cross-subnet, since the rooms are separate subnets) to values that tolerate the 1 Gb heartbeat's real latency - e.g., CrossSubnetDelay 2000, CrossSubnetThreshold 10 - so a transient blip does not isolate the node. Also move the heartbeat to the 10 Gb network; a saturated 1 Gb link during failover can delay heartbeats and cause spurious isolation.
2. The SAN lost power with NODE01. What does this reveal about the HA design, and what change would meet the 15-minute RTO without adding a third compute node?
Read the worked answer
Reveals a single point of failure. The cluster is highly available for node failure but not for storage failure. A single SAN in one room means the 15-minute RTO is unachievable regardless of node count - the third node the business is considering would not help, because all nodes would lose the same LUNs.
Change. Separate storage from the failure domain: either (a) replicate the SAN to a second array in the other room with synchronous replication and present both as cluster disks with Storage Replica in stretch-cluster mode, or (b) move to Storage Spaces Direct with at least two nodes per room (four total) - but that exceeds budget, so option (a) is the fit. With synchronous Storage Replica between the two rooms, a SAN outage in one room leaves the replica in the other room available; the cluster fails over to the surviving copy within minutes, meeting 15 minutes.
Also required. Place the SAN and its replica on independent power feeds/UPS, and put the witness outside both rooms. Document that the 15-minute RTO now depends on replication lag being near zero - synchronous mode enforces that.
3. A colleague proposes removing the heartbeat network entirely and relying on the iSCSI network for cluster communication. Evaluate this.
Read the worked answer
Reject it. Cluster communication and storage traffic on the same network creates contention: iSCSI bursts (especially during CSV redirection or a failover storm) can delay or drop cluster heartbeats, which is exactly the condition that produced the spurious quarantine. The heartbeat is low-bandwidth but latency-sensitive; iSCSI is high-bandwidth and bursty.
Better configuration. Keep a dedicated network for cluster communication, but make it redundant - two NICs teamed or two separate subnets - and ensure it is not the same physical switch path as iSCSI. Set the cluster network roles explicitly: Cluster and Client for the management network, Cluster Only for the heartbeat, and None for iSCSI so the cluster does not use storage NICs for heartbeats. Verify with Get-ClusterNetwork. If the 1 Gb heartbeat link is the only path, a single cable or switch failure isolates a node, so redundancy matters more than removing the link.