HA Admission Control That Fails the VMs It Was Supposed to Protect
An 8-host cluster runs 240 VMs. Each host has 2 sockets, 64 cores, and 1 TB of RAM. HA is enabled with 'Cluster resource percentage' admission control set to 25% and 'host failures to tolerate' set to 1. DRS is fully automated. Cluster-wide memory usage is 62% and CPU is 30%. Forty business-critical VMs carry a 32 GB memory reservation each, totalling 1.4 TB of reservations; the remaining VMs have small or zero reservations. Last month a host failed. HA restarted 22 VMs, but 6 of the critical VMs stayed powered off with 'Insufficient resources to satisfy the configured reservation'. Separately, a mandatory VM-VM anti-affinity rule between two domain controllers was found violated after the failover and DRS never corrected it.
1. The cluster shows about 38% free memory, yet six VMs could not restart. Explain the discrepancy and what HA was actually measuring.
Read the worked answer
The discrepancy
- Free memory is not what HA checks. HA checks whether the remaining hosts can satisfy the reservations of the VMs it needs to place. A VM with a 32 GB reservation needs 32 GB of unreserved memory on one host, contiguously, not a cluster-wide average.
- After the host failure, the cluster lost 1 TB of the 8 TB total. Admission control at 25% had already reserved a quarter of the cluster for failover, but the 25% figure was computed from aggregate resources, not from the shape of this workload's reservations.
- Six 32 GB reserved VMs need to land on hosts whose unreserved memory still exceeds 32 GB each. If DRS had packed reserved VMs densely, individual hosts run out of unreserved memory even while the cluster total looks healthy. That is a per-host placement problem, and HA has no lever to fix it.
What to check
- Per-host unreserved memory, not cluster averages.
- Current admission control status and the exact message in the VM's event log.
- Whether the reservations are genuinely required - a 32 GB reservation on a VM that does not need guaranteed memory is the root cause, not HA.
Reasoning - reservations are a hard admission test; percentages are a soft planning figure. Any policy that protects reservation-heavy workloads must be sized against the largest reservation, per host.
2. Recommend an admission control policy for this cluster and justify the choice against the alternatives.
Read the worked answer
Recommendation - a slot policy with a custom slot size, or a percentage deliberately sized to the reservation profile. Work through it:
- Why the current percentage policy underperforms - it reserves a slice of aggregate CPU/memory and does not reason about 32 GB reservations or about fragmentation across hosts. It permits a placement that is arithmetically compliant and operationally useless.
- Slot policy with custom slot size - set the slot size to the largest reservation profile (memory 32 GB, plus the largest CPU reservation). This forces the maths to reflect the worst-case VM, which is exactly the workload that failed. Cost: it is conservative and may block power-ons on a cluster with heterogeneous VM sizes.
- Percentage policy, corrected - only defensible if you compute the percentage from the actual failover requirement (worst-case host's workload plus reservations) rather than leaving the default, and if DRS rules prevent reserved VMs from being packed onto one host.
- Dedicated failover hosts - strongest guarantee for the 40 critical VMs if you can afford to keep a host idle; usually rejected on cost here.
Also do
- Reduce reservations that are not justified - this is the cheapest fix and improves both HA and DRS behaviour.
- Create a VM-VM anti-affinity rule for the reserved critical VMs so DRS keeps them spread, giving HA somewhere to place them.
- Re-test with a host in maintenance mode before the next real failure.
Reasoning - pick the policy that models your worst-case VM, then validate it by simulating a failure. A policy you have not tested is a hope.
3. Why was the domain controller anti-affinity rule violated after failover, and what would you change?
Read the worked answer
Why it happened
- HA failover placement is not DRS placement. When a host dies, HA must restart VMs immediately on whatever hosts have capacity; it does not perform a full DRS-style balancing pass first. Anti-affinity is honoured where HA can, but under resource pressure it will not leave a critical VM powered off to preserve a rule.
- After the restart, DRS should be able to correct the violation with a vMotion - but a mandatory anti-affinity violation blocks further operations on those VMs rather than being silently fixed, and if no host can accept one of the pair, DRS has nowhere to move it. That is a capacity problem disguised as a rules problem.
- Rule type matters: 'should' rules are recommendations DRS balances against; 'must' rules are hard constraints that can block placement entirely.
What to change
- Verify the rule is actually enabled and applied to both VMs by name/UUID, not to a stale inventory path.
- Confirm there is genuine spare capacity on a third host for one of the pair - add a VM-VM 'should' rule as a soft guard if a 'must' rule creates a deadlock.
- For a pairing this critical, consider separating them across clusters or datacenters rather than relying on an intra-cluster rule; an anti-affinity rule cannot protect against a whole-cluster outage.
- Re-run a failover test and observe whether DRS remediates within a few cycles.
Reasoning - HA answers 'can I restart it', DRS answers 'where should it live'. Rules only work when capacity exists to satisfy them, so validate the rule and the capacity together.