GPU Operator Rollout Breaks MIG Scheduling
You manage a 16-node Kubernetes cluster with NVIDIA GPU Operator 23.9, driver 535, CUDA 12.2, and containerd. Eight nodes are A100 80GB systems with MIG enabled and advertise nvidia.com/mig-1g.10gb. You roll the operator to driver 550/CUDA 12.4. After the rollout, several MIG nodes report Allocated, but new pods stay Pending with 'nvidia.com/mig-1g.10gb: 0/7'. The device-plugin logs show 'MIG profile not found'. On nodes with UEFI Secure Boot enabled, the driver pod crashes and dmesg reports that an unsigned module was rejected. Non-MIG nodes are healthy. You must diagnose the failure, define a safe canary and rollback procedure, and validate that MIG scheduling works again.
1. Diagnose why MIG pods are Pending after the driver upgrade. What evidence do you collect from the node, the operator, and the device plugin?
Read the worked answer
Check whether the new driver actually loaded: run nvidia-smi on the node and inspect the driver pod logs. On Secure Boot nodes, dmesg rejecting an unsigned module explains why the driver is absent; without the driver, the device plugin cannot discover GPUs or create MIG instances. On nodes where the driver loaded, verify MIG mode and profiles with nvidia-smi -L, nvidia-smi mig -lgip, and nvidia-smi mig -lgi. The operator may have reset MIG configuration during the upgrade, so no 1g.10gb instances exist and the device plugin reports 'MIG profile not found'. Check node labels such as nvidia.com/mig.config, the device-plugin ConfigMap, and whether the MIG manager pod is running. Correlate all three sources before changing anything.
2. Define a safe rollout and rollback procedure for the driver and operator upgrade, including Secure Boot handling.
Read the worked answer
Use a canary: cordon and drain one MIG node, upgrade the operator or driver there, reboot, and verify nvidia-smi, MIG profiles, device-plugin resources, and a test pod before touching other nodes. For Secure Boot, either enroll a signed driver or follow an approved policy to disable Secure Boot for GPU nodes; do not leave unsigned modules failing intermittently. Keep the previous operator version and driver image available, and know the Helm rollback command. If the canary fails, roll back the operator, restore the previous driver, reboot, and confirm MIG resources return before uncordoning. Only then proceed node by node with drain and validation at each step.
3. What validation proves MIG scheduling is healthy again, and what monitoring would prevent recurrence?
Read the worked answer
Confirm nvidia-smi -L shows the expected MIG UUIDs and nvidia-smi mig -lgip lists 1g.10gb. Verify the node advertises nvidia.com/mig-1g.10gb: 7 and that the device plugin is healthy. Submit a test pod requesting one MIG resource, run nvidia-smi inside it, and confirm it is pinned to a MIG UUID. Run a small CUDA workload and check DCGM for XID, ECC, and thermal events. Add alerts for driver pod restarts, missing MIG resources, and Secure Boot module rejection, and gate future operator upgrades on a canary node with the same MIG profile.