AOS Upgrade with GPU Passthrough VMs Pinned to Two Hosts

The core constraint is that VMs with PCI passthrough cannot live-migrate, so AHV maintenance mode will block until they are powered off or the host is evacuated by other means. Plan around that instead of forcing it.

Sequence:

1. Pre-checks. Run LCM prechecks, confirm AOS/firmware compatibility, snapshot or back up Prism Central and critical VM configurations, and verify cluster health. Confirm the radiology downtime window in writing and have clinical scheduling acknowledge it.
2. Upgrade nodes 1-3 first. They have no pinned VMs, so standard maintenance mode and live migration should work. Upgrade them one at a time and verify cluster health between nodes. Do not start until each node is fully back in service.
3. Handle node 4. Move all migratable VMs off node 4. The pinned radiology VM and possibly the appliance must be powered off during the approved 20-minute window. Use a maintenance-mode runbook that explicitly lists which VMs will power off, and have a named approver. Power on and validate the radiology VM before touching node 5.
4. Handle node 5. Repeat in a second approved window, because the 20-minute approval is unlikely to cover both GPU hosts plus remediation. Do not assume both can be done in one four-hour window if validation is included.
5. Firmware on nodes 1-3 can follow the AOS upgrade or be bundled only if the vendor matrix supports it.

If clinical downtime cannot be split, the alternative is to defer node 5 and document partial upgrade risk. Never let an engineer power off a clinical VM outside the approved runbook.

These are different risks and should not be treated as one change.

Overcommit 6:1 to 4:1. A 6:1 ratio on a dense VDI pool can work if the workload is bursty and idle most of the day, but it is fragile during login storms, antivirus scans, or host loss. Dropping to 4:1 reduces contention and improves tail latency, but it may not be possible on the existing five nodes without adding capacity or reducing VDI density. The right method is to measure CPU ready time, host CPU utilization at peak, and the number of vCPUs per physical core, then set a ratio that survives N+1 host loss. If 4:1 cannot fit, the honest answer is that the cluster is undersized for the stated latency target, not that the ratio should stay at 6:1 because it fits.

Removing affinity from the appliance. A VM with a physical NIC passed through is bound to the physical PCI device on a specific host. Removing the affinity rule does not make it migratable; it only removes the scheduler guardrail and can cause placement failures or unexpected behavior. The appliance should stay pinned unless the physical NIC is replaced with a virtual NIC or SR-IOV that supports migration. For the GPU VMs, the same logic applies: the affinity reflects hardware, not preference.

The better change is to inventory which VMs are truly pinned by hardware and document them as a protected set with explicit maintenance procedures. Then size the remaining migratable workload so N+1 maintenance is feasible.

The failure was procedural, not technical. Controls should make the unsafe action visible and hard to perform.

• Runbook with explicit VM list. Each host's maintenance plan must name every VM that will be powered off, the approver, and the clinical window. Generic 'enter maintenance mode' instructions are insufficient.
• Role separation. Engineers running LCM should not have permission to power off clinical VMs outside an approved change. Use Prism Central RBAC and project scoping; require a second approver for power operations on the clinical project.
• Pre-flight check in LCM. Before the window, run a script or report that identifies VMs with passthrough devices, affinity rules, or other migration blockers, and attach it to the change ticket.
• Maintenance-mode guardrail. Treat any AHV maintenance-mode task that waits on a powered-off VM as a stop condition, not a prompt to force it. Document the expected behavior so on-call staff do not improvise.
• Change freeze during clinical hours. Enforce a calendar-based freeze for the radiology project from 07:00 to 19:00, with an exception process.
• Post-change validation. Require VM power-state and application checks before moving to the next node, and capture evidence in the ticket.
• Incident review. The prior outage should have produced a corrective action; verify it was closed and tested, not just documented.