Operations, Monitoring, and Lifecycle Management for AI Environments
Operations, Monitoring, and Lifecycle Management for AI Environments
AI environments add GPU health, driver compatibility, and capacity pressure to normal Nutanix operations. The exam tests monitoring, upgrades, and lifecycle discipline.
Core practices
- Use Prism Central for cross-cluster alerts, capacity, and VM/container visibility; use NCC for cluster health checks.
- Monitor GPU utilization, memory, temperature, ECC errors, and vGPU licensing; correlate with workload latency.
- Use Life Cycle Manager (LCM) for AOS/AHV/firmware updates; follow the NVIDIA driver and GPU Operator compatibility matrix before upgrading.
- Use Nutanix Cloud Manager (NCM) for cost visibility, rightsizing, and governance.
- Plan capacity from GPU memory and node pool headroom, not just CPU/RAM.
- Maintain runbooks for model rollout, rollback, and data restore.
Worked example
A GPU node pool shows pods pending while Prism shows idle GPUs. Check GPU Operator/device plugin health, node taints, and pod resource requests. If the device plugin is down, the scheduler sees zero allocatable GPUs even though hardware is present. Fix the operator, then recheck allocatable resources.
Traps
- Upgrading GPU drivers or Operator before verifying NKP/NCI compatibility.
- Monitoring only cluster CPU and missing GPU memory exhaustion.
- Ignoring NCC warnings before an AI rollout.
- Treating LCM as a workload deployment tool.