Maintenance Mode on a Nearly Full Six-Node Cluster
A six-node AHV cluster runs a single storage container with RF2. Each node contributes 8 TB of raw capacity, so usable capacity is roughly 24 TB, and the container is 80% full with 19.2 TB of VM data. Erasure coding is off. One node has an intermittent DIMM error and the hardware vendor needs it powered off for about eight hours, starting in two hours. No spare node is available. The line-of-business owner accepts a hardware window but will not accept the VMs being down for the whole period. In Prism Element the engineer is about to confirm Enter Maintenance Mode and must choose between 'Ensure resiliency' and 'No data movement'. Curator reports no rebuild in progress and all extents healthy, and the network team confirms inter-host traffic has ample headroom.
1. Which maintenance mode should be selected, and what actually happens under each option on this specific cluster?
Read the worked answer
Do the capacity arithmetic before touching the toggle.
- With the node out: 5 nodes x 8 TB = 40 TB raw.
- 19.2 TB of data at RF2 needs 38.4 TB of raw space.
- That is 96% full, before metadata reserve and any Curator headroom.
'Ensure resiliency' evacuates the node's data so the cluster is fully protected during the window. Each node holds roughly 19.2 / 6 = 3.2 TB, so the volume of data to move is not the problem - the destination is. Five nodes cannot absorb those replicas without pushing the cluster to the edge of full, where Curator throttles, extents fail to allocate, and a single additional failure becomes a data-loss event. The mode is not viable here.
'No data movement' shuts the node's CVM down and serves its data from the surviving replicas. Because RF2 already holds a second copy, the data stays available; the cluster simply runs at N+0 for eight hours. VMs on the host must be moved off it by AHV live migration, which is a separate action from the storage decision.
Recommendation: select 'No data movement', pre-migrate the VMs, and treat the window as an accepted single-failure-exposure event - or add capacity first (see next task).
2. The CISO refuses to accept a single-failure-exposure window of that length. What is the smallest change that makes 'Ensure resiliency' safe, and how do you justify it?
Read the worked answer
The blocker is free capacity on the remaining nodes, not evacuation time.
Add one node (7 total):
- 7 x 8 TB = 56 TB raw; with the failed node out, 48 TB raw.
- 38.4 TB of RF2 data against 48 TB = 80% used, which leaves room for the evacuated replicas plus metadata and Curator slack.
- The rebuild destination must be in the same container and the same cluster; a node in a different cluster does not help.
Alternatives worth stating, and why they rank lower:
- Reduce RF to 1 - never acceptable. It multiplies the blast radius and does not solve the evacuation problem; going back to RF2 later requires a full re-protection cycle.
- Delete or migrate data off-cluster first - viable if a low-value container exists, but it must be measured against the evacuation need, not guessed.
- Postpone and replace the DIMM under a shorter window - legitimate if the hardware vendor can do a 2-3 hour swap, since 'No data movement' exposure scales with duration.
Justification to the CISO: quantify the exposure (8 hours at N+0 across ~19 TB of protected data), show the math that resiliency mode cannot complete safely today, and present node-add as the only option that restores the intended RF2 posture during the outage.
3. If the team proceeds with 'No data movement', what must be verified or watched during the eight hours, and what is the abort criterion?
Read the worked answer
Before the window:
- Confirm no second node has a predictive disk failure or a failing NCC check; check the health page and disk status host by host.
- Check AHV HA / HA reservation. At N+0 a further host loss may not have anywhere to restart the VMs, so know in advance which workloads would be stranded.
- Confirm replication/snapshot jobs do not overlap the window; a heavy snapshot burst competes for the same I/O.
During the window, watch:
- Data resiliency status (should stay 'reduced' only for the intended reason, and must not degrade further).
- Container usage trend - any unexpected growth eats the thin margin.
- Any new disk or CVM alert on the remaining four nodes; a second fault is the difference between 'degraded' and 'lost'.
Abort criteria (state them explicitly in the change record):
- Any second node or disk failure.
- Cluster utilization crossing the agreed ceiling.
- Any CVM not responding, or VMs unable to migrate.
Abort action: bring the node back online immediately, or shut down non-critical VMs to release pressure. After the window, verify resiliency has returned to full and no rebuild is stuck.