RF3 Request on a Five-Node Cluster That Cannot Absorb a Node Failure
A five-node NX cluster runs AOS with AHV. Each node has six 1.92 TB SSDs, giving roughly 57.6 TB of raw cluster capacity; Prism Element reports about 26.5 TB of usable capacity at RF2 after metadata, CVM, and curator reserves. All existing containers use RF2 and average 55% consumption. The database team wants a new container, CriticalData, with replication factor 3 to hold 12 TB of data. They argue that 12 TB x 3 = 36 TB, which is less than the 57.6 TB of raw capacity, so it must fit. The change window is Friday night and they want the container created and seeded in one pass.
1. Does the 12 TB RF3 container fit, and what is the real risk if you create it as requested?
Read the worked answer
The arithmetic fits but the cluster does not. Raw 57.6 TB minus 36 TB consumed by an RF3 copy of 12 TB leaves about 21.6 TB of raw space, roughly 10.8 TB of effective RF2 space.
That headroom is smaller than one node's worth of data (about 11.5 TB raw). RF3 means each extent lives on three of the five nodes; if one node is lost, Curator must rebuild those extents onto the surviving nodes. With only ~21.6 TB raw free, the rebuild has nowhere to land, so the cluster drops into a degraded, non-rebuildable state and a second failure becomes a data-loss event rather than an availability event.
Also check whether RF3 is even the right tool: RF3 protects against two simultaneous failures but costs 3x, and it is a poor fit for a cluster sized for RF2. Create the container only after either adding a node or shrinking the requirement.
2. The customer insists on RF3 for all 12 TB. Give two concrete options and state the trade-off of each.
Read the worked answer
Option 1 - add a sixth node before the window. Adding a node raises both raw capacity and failure-domain count, which is what RF3 actually needs. With roughly 69 TB raw and 36 TB still consumed by the RF3 container, free raw space (~33 TB) exceeds a single node's data, so a rebuild can complete. Trade-off: hardware lead time and a rebalance before the window.
Option 2 - split by data class. Put a small subset (for example, the 2 TB of truly irreplaceable data) in an RF3 container and keep the remaining 10 TB in an RF2 container with erasure coding enabled where the AOS version and cluster size support it. Trade-off: two containers to manage and an application-side decision about which data is which; also verify EC support in your AOS release before promising the space savings.
What you should not do is promise RF3 on the current five nodes and hope the Friday window works out. If the customer cannot accept either option, state in writing that the cluster will be unable to rebuild after a single node failure.
3. How would you validate the outcome after the change, using Prism Element data rather than assumptions?
Read the worked answer
After the container is created, open the Storage dashboard and check capacity by container and by RF, not just the aggregate usable figure. Confirm:
- The RF3 container's logical data versus consumed physical space shows close to a 3x multiplier.
- Free space at the cluster level is still larger than the largest node's used data, which is the practical test for rebuild capability.
- The Health page shows no capacity or resiliency alerts, and no 'rebuild capacity' warning appears after simulating what a node loss would consume.
Then take a baseline snapshot of these numbers so the next change window can be measured against it. If free space after the change is at or below one node's used data, treat the change as a failure even if no alert fired.