Change Window Collision: A CVM Fails Mid-Expansion
An engineer is working a four-hour change window on a four-node NX-3060 cluster running AOS 6.7 on AHV, RF2, 180 production VMs, 61% storage utilization and one Prism Central VM. The plan is to add two new nodes, enable EC-X, then upgrade AOS to 6.8 with LCM. At 02:10, while the first new node is being commissioned, the CVM on node 3 stops answering. The AHV host and its VMs remain powered on, Prism marks node 3 as down, and alerts fire for 'CVM not responding' and 'replication factor degraded'. The two new nodes finish imaging and join, but each shows only a few GB used. No replication or backup target exists for these VMs, and the last successful backup is nine days old. The on-call engineer argues the upgrade can proceed because 'the cluster is still serving I/O and Prism is up.'
1. Explain what is still working and what has actually been lost from a resiliency standpoint, naming the AOS services involved and why the VMs on node 3 are at risk even though the host is powered on.
Read the worked answer
What still works. VMs on the three healthy nodes keep running because each host's I/O is served by the Stargate process inside its local CVM, and cluster metadata lives in the distributed Cassandra ring, which retains quorum with one of four nodes down. Prism Element and Prism Central stay reachable, so the UI looks healthy at a glance.
What is actually lost.
- Every extent whose second copy lived on node 3 now has a single available copy. RF2 has become effectively RF1: another node failure or an unreadable disk in that set risks data loss, not just downtime.
- No rebuild can complete while node 3 is absent; the surviving copy is the only source.
- Curator cannot rebalance or reclaim space from a node it cannot see, so those extents stay pinned.
Why node 3's VMs are at risk. The host being powered on does not mean storage access survives. Its VMs depend on the local Stargate for I/O; with the CVM unresponsive they will stall on I/O and are likely to be restarted elsewhere by HA, which is a workload event the engineer has not accounted for. Expect hung applications on that host first, alerts second.
2. Should the engineer proceed with EC-X and the LCM upgrade now? Give the order of operations and the conditions that must be met before each step.
Read the worked answer
No. Both remaining steps are cluster-wide, I/O-heavy operations that assume a healthy, fully redundant cluster.
- EC-X now: erasure coding requires a healthy RF2 cluster with enough nodes and free capacity to lay down 4+1 stripes. Enabling it while one node is down forces a large rewrite of data that currently has only one copy, and the capacity calculation is wrong because a node's worth of space is unavailable.
- LCM upgrade now: the pre-checks (NCC, free space, CVM health, Prism Central compatibility) will not pass cleanly, and a rolling upgrade that reboots CVMs one at a time on a cluster already missing a CVM removes the last safety margin.
Correct order:
- Recover or evacuate node 3 - check IPMI/console, try restarting the CVM, verify network and disk health before anything else.
- Wait until Prism reports the cluster's resiliency state as OK and no 'replication factor degraded' alerts remain.
- Run NCC to a clean result.
- Enable EC-X only if the workload and capacity justify it, and evaluate it as its own change.
- Do the AOS upgrade in a separate window, then confirm health before adding further changes.
3. Why do the two new nodes show almost no used capacity, and what should the engineer tell the capacity owner about when utilization will fall below 61%?
Read the worked answer
New nodes join the storage pool but do not trigger a bulk copy of existing data. Nutanix places data by where it already lives; Curator rebalances only when the imbalance or capacity pressure justifies the I/O cost, and it will not move extents around merely because empty nodes arrived. What fills new nodes first is new writes and newly created vDisks, plus the rebuild or re-protection traffic once node 3 is dealt with.
So the honest answer to the capacity owner is:
- Reported utilization may barely move for days, and it can temporarily rise because re-protection and rebalancing consume space.
- Raw capacity is not usable capacity. Each node reserves space for metadata and the CVM, so a two-node addition does not give you the sum of the drive sizes.
- If the goal is headroom for growth rather than a lower percentage, the cluster already has it; the percentage is a placement artifact, not a free-space emergency.
Do not promise a specific new percentage until rebalancing has completed and EC-X, if enabled, has finished converting data - both change the math.