RF3, Block Awareness, and a Maintenance Window
An 8-node Nutanix cluster runs 260 VMs for a logistics company. The nodes are spread across two blocks of four, and the cluster is configured with RF3 and block awareness. Last Tuesday a PDU failure took down nodes 3 and 4, both in block B; the cluster stayed up and all VMs remained available. The operations lead now wants to place node 5 into maintenance for a disk firmware update during the same week, arguing that RF3 tolerates two failures and the PDU has been replaced. The cluster is at 68% storage utilization, and a Curator scan completed 14 hours ago. The change window is four hours. The vendor compatibility matrix lists the firmware as 'recommended' but not urgent. No spare nodes are available, and the cluster has two 10 GbE links per node.
1. Assess the operations lead's claim that RF3 tolerates two failures, so the maintenance is safe.
Read the worked answer
The claim is too simple. RF3 means three replicas and tolerance of two failures only when the failures are in independent fault domains. With two blocks, the cluster cannot place three replicas on three distinct blocks; some vDisks may have two replicas in one block and one in the other. Losing the block with two replicas leaves a single copy, which is available but at RF1. The PDU event already demonstrated a block-level failure; replacing the PDU does not prove the cluster's current placement or free space.
Before maintenance, check the actual data resiliency status, not just the RF setting. Node 5 maintenance removes one node; if placement is 2+1 across blocks, taking that node down can leave some vDisks with one copy. A second failure during the window could cause unavailability. Since the firmware is only 'recommended', the safer judgement is to defer until the cluster is fully resilient and the previous event's root cause is closed, or to perform one node at a time with explicit RF1 monitoring.
2. List the pre-checks and the sequence you would require before approving the maintenance.
Read the worked answer
Require these pre-checks and sequence:
- Resiliency: Prism/NCC shows all vDisks healthy at the expected RF; no degraded or rebuilding vDisks; no outstanding alerts.
- Capacity: enough free space to rebuild or absorb the maintenance node; 68% utilization is not automatically safe-calculate per-node capacity and rebuild headroom.
- Fault domains: confirm block placement and that no other maintenance or failure is active.
- Firmware path: vendor matrix, upgrade order, and rollback; one node at a time, waiting for full resiliency before the next.
- Backups: recent restorable backup, especially for the database VMs.
- Change window: four hours may be enough for one node; do not chain nodes.
During the window, monitor latency, rebuild status, and CVM health. Stop if the cluster drops below the agreed resiliency threshold.
3. What is the risk if the cluster is at RF1 during the maintenance, and how would you mitigate it?
Read the worked answer
At RF1, every vDisk has only one remaining copy, so a second independent failure-another node, a disk, or a block-can make data unavailable or lost. Performance also degrades because there is no local second copy for reads and rebuilds compete for resources. Mitigate by not starting maintenance until resiliency is restored, taking a verified backup, freezing other changes, and monitoring continuously. If maintenance must proceed, get explicit risk acceptance and define an abort criterion: if any vDisk goes degraded or a second node alerts, stop and restore redundancy. Adding a temporary node or waiting for the firmware's urgency to be reclassified are alternatives.