Firmware window on a five-host cluster at N+1
A five-host virtualization cluster runs 84 general-purpose VMs at 4 vCPU and 20 GB RAM with memory reservations set to 100%, plus six 64 GB, 8 vCPU SQL VMs whose replicas live at a DR site in another region. Each host has 32 physical cores and 512 GB RAM. HA admission control is configured as N+1. A firmware and driver update must be applied to all five hosts. Each host needs roughly 75 minutes in practice to drain, update, reboot, and rejoin, and past drains have run over that estimate. The change window is four hours long and is granted once. There is no spare host. Last month a patching drain triggered an insufficient-resources alarm.
1. Can all five hosts be updated in this window while keeping N+1 the whole time? Show the arithmetic.
Read the worked answer
No on both counts - capacity and time.
Capacity (memory is the binding resource, not CPU):
- Committed memory:
84 x 20 GB = 1680 GBplus6 x 64 GB = 384 GB= 2064 GB. - During any single host's maintenance, four hosts remain:
4 x 512 GB = 2048 GB. 2064 > 2048by 16 GB before you count VM memory overhead, so admission control will refuse to place every VM on four hosts. N+1 is already broken at the first drain - which is exactly what the alarm reported last month.- CPU is not the constraint:
84 x 4 = 336plus6 x 8 = 48= 384 vCPU on4 x 32 = 128physical cores, a 3:1 overcommit that is normal for this workload.
Time:
5 hosts x 75 min = 375 min = 6 h 15 min, against a 240-minute window. Serialization is mandatory: you cannot drain two hosts at once without breaking N+1 by two.- Even at 60 minutes each you would need 300 minutes, still over.
Conclusion: the request as written is not achievable. Something has to give: committed memory, the number of hosts per window, or the N+1 guarantee during the change.
2. Propose an execution plan the change board will approve, and state what you are trading away.
Read the worked answer
Target: keep N+1 for every minute of every window, and stop buying risk with the DR site.
- Free memory in this site. Power off (not migrate) the six 64 GB SQL secondaries here; the DR replicas become the recovery point for the duration. Committed memory falls to
1680 GB; on four hosts that is1680 / 2048 = 82%with about 368 GB of headroom. N+1 now genuinely holds.- Trade: the SQL estate runs without a local secondary. Get written sign-off from the database owner and confirm DR replication is current and DR can be promoted before you start.
- Split into two windows. Window 1: three hosts,
3 x 75 = 225 min, fits inside four hours with slack. Window 2: two hosts, 150 min. Trying to force all five into one window is what pushes people into skipping drain checks. - Order of work. Update vCenter/management components first, confirm the cluster still functions, then hosts strictly one at a time. Fully validate the first host - storage paths, uplinks, HA agent state, driver versions - before touching the second. A bad driver image caught on host one costs one host; caught on host four it costs the cluster.
- Do not touch the DR site in the same window. It is the fallback while local redundancy is reduced.
- Restore the SQL secondaries and re-enable the full reservation profile only after the last host is validated, then re-run the admission control calculation to prove N+1 is back.
Traded away: time (two windows instead of one) and temporary loss of local SQL redundancy. Both are visible and reversible; the alternative is an unplanned outage in the middle of a change.
3. What pre-flight checks and stop criteria do you insist on before the first host goes into maintenance mode?
Read the worked answer
Pre-flight:
- Check every firmware and driver version against the vendor's compatibility matrix, and check the hypervisor build against that matrix - a firmware that is newer than the certified pairing is a common cause of 'it booted but lost its storage'.
- Confirm out-of-band console access works on all five hosts. If a host fails to boot after a NIC or HBA firmware flash, in-band access is gone and the BMC is the only way back.
- Confirm a tested restore of at least one VM within the last 30 days, and that DR replication lag is near zero.
- Record current versions, multipath state, and the HA admission control settings so the rollback point is documented, not remembered.
- Verify that the previous firmware images are available and that the vendor supports downgrade for each component. Some controllers and NICs cannot be rolled back, which is precisely why you validate on one host first.
Stop criteria - any one of these halts the window:
- A host fails to rejoin the cluster or HA agent does not come back healthy within a defined time (for example 15 minutes past expected).
- Any storage path loss or datastore inaccessible event, even transient.
- Admission control or any other alarm indicating N+1 is no longer satisfied.
- Drain time exceeding the plan by more than 20%, which means the window math no longer holds for the remaining hosts.
On a stop: leave the affected host out of the cluster, leave the remaining hosts on the old version, restore the powered-off secondaries, and rebook. A half-applied fleet with an unknown fault is worse than an old firmware fleet.