N+2 Resilience on a Cluster That Is Already Overcommitted
A financial services firm runs one 6-host cluster. Each host has 2 x 16 cores and 512 GB RAM. HA is enabled with 'host failures the cluster tolerates = 2' using the cluster resource percentage policy and default rounding. The cluster hosts 150 VMs: 100 VMs sized at 12 GB with no memory reservations, and 50 VMs sized at 32 GB each carrying a full 32 GB memory reservation because of a vendor requirement. Nothing is currently overcommitted at the host level and no alarms are firing. The application team now wants to add a new tier of 40 VMs, each 64 GB and fully reserved, into the same cluster. Storage is on a separate array and is not a constraint. Management will not accept less than N+2 in this cluster.
1. Is the cluster actually N+2 compliant today? Show the arithmetic and explain why nobody has noticed the gap.
Read the worked answer
Arithmetic (memory-only, since memory is the binding resource)
- Cluster total: 6 hosts x 512 GB = 3,072 GB.
- Failure percentage: 2 / 6 = 33.3%, which HA rounds up to 34%.
- Capacity reserved for failover: 0.34 x 3,072 = ~1,044 GB.
- Admissible memory: 3,072 - 1,044 = ~2,028 GB.
Committed today
- 100 x 12 GB = 1,200 GB
- 50 x 32 GB = 1,600 GB
- Total = 2,800 GB
2,800 GB committed against ~2,028 GB admissible: the cluster is roughly 770 GB over its own N+2 budget. It is not N+2 compliant today.
Why no one noticed
HA admission control only blocks new power-ons, registrations and migrations that would breach the policy. It never evicts or refuses to run already-running VMs. Everything booted before the reservations accumulated keeps running happily. The failure mode is silent until a host actually dies - at which point the second host failure leaves roughly 2,048 GB of capacity to restart 2,800 GB of committed memory, and the reserved 32 GB VMs simply will not all come back.
Also worth showing them
A slot-based policy would fail even harder: slot size is driven by the largest reservation plus overhead, so a ~33 GB slot against 3,072 GB yields only about 93 slots for 150 VMs. The percentage policy is the less pessimistic of two failing options here - the design problem is the committed memory, not the policy.
2. The 40 new 64 GB VMs must run in this cluster with N+2 preserved. What is the minimum number of same-model hosts, and what two alternatives preserve N+2 with fewer hosts?
Read the worked answer
Target committed memory
2,800 GB existing + (40 x 64 GB = 2,560 GB) = 5,360 GB, before per-VM overhead of roughly 1 GB each. Plan against ~5,400 GB.
Sizing for N = hosts
With tolerance of 2, admissible capacity is 512N - 1,024 GB (the two failed hosts' 512 GB each).
- Need: 512N - 1,024 >= 5,360
- 512N >= 6,384
- N >= 12.5, so 13 hosts is the mathematical floor.
Check 13: 6,656 - 1,024 = 5,632 GB admissible, versus ~5,400 GB with overheads. That is under 5% headroom on a cluster expected to absorb a new application tier - too tight to sign off. Design at 14 hosts (7,168 - 1,024 = 6,144 GB admissible).
Alternatives with fewer hosts
- Attack the reservations. The 32 GB full reservations are the real cost: they convert 1,600 GB of configured memory into 1,600 GB of guaranteed memory. If the vendor accepts a 50% reservation or a sizing guarantee instead, committed memory drops sharply and the host count falls with it. Get the requirement in writing.
- Split the cluster by tier. One cluster at N+2 for the core 250 GB workload, a second cluster at N+1 for the new 40 VMs. This also fixes the rounding penalty: 2/6 wastes 34% of capacity, while a smaller dedicated cluster at N+1 wastes far less per host.
Do not accept a CPU-based justification for shrinking hosts - CPU is not what blocks this design.
3. Which admission control policy would you standardise on for this environment, and what would you monitor so a design drift like this is caught in weeks rather than at failover?
Read the worked answer
Policy choice: cluster resource percentage
- Slot policy derives slot size from the largest reservation plus overhead. With 64 GB reserved VMs, every slot becomes ~65 GB, so capacity is computed as if the whole cluster were made of the biggest VM. It is simple and conservative to the point of being unusable here.
- Percentage policy is capacity-based and scales sensibly with host count. Set the percentage explicitly rather than letting the wizard derive it, and document why: 34% is the N+2 number for 6 hosts, but it must be recalculated after every host addition, because 2/14 rounds to 15%.
- Note the trap: percentage policy counts configured memory for VMs without reservations. That is deliberately conservative, and it is why the reservation strategy is a design decision, not a per-VM detail.
Monitoring that catches drift
- A weekly report of cluster committed memory (reservations, falling back to configured) versus admissible capacity, with a 10% early-warning threshold - not a 100% threshold.
- An alarm on VM memory reservations above a size threshold, so a new reservation raises a change review instead of silently consuming the failover budget.
- Re-run the calculation as part of every capacity request; treat host count and reservation policy as inputs to a formula, not folklore.
- Periodically prove it: place a host into maintenance mode during a low-risk window and observe whether HA can evacuate and restart within policy. A failover that has never been rehearsed is a hypothesis.