RF3 on a Four-Node Cluster: Audit Requirement vs. Physics
Skyline Logistics runs a 4-node all-flash cluster in one data center: each node has 6 × 3.84 TB SSDs, 2 × 16-core CPUs and 512 GB RAM, and hosts 210 production VMs. Prism Element shows 28.4 TB of 40.6 TB usable consumed (70%), with steady growth of about 1.2 TB per month. A new auditor has told the CIO that the customer order database must survive the simultaneous loss of any two nodes, and security has asked the team to "turn on RF3" before the audit in three weeks. The platform team has one identical node still in its crate on the loading dock. During last month's maintenance window, a node was powered off for four hours for firmware work and the cluster kept running, so the team believes the resilience requirement is already met. The CIO wants a written recommendation, not a purchase order, including what happens during the next single node failure if nothing changes.
1. Can RF3 be enabled on this cluster as it stands? Show the arithmetic that determines the answer and the minimum viable change.
Read the worked answer
No - RF3 is not even selectable on four nodes. Nutanix requires a minimum of five nodes before a cluster can be set to RF3, because three replicas plus rebuild activity only make sense when two nodes can be lost and the surviving set can still hold every extent.
Adding the crated node makes RF3 allowed, but not advisable:
- 4 nodes at RF2: raw ≈ 4 × 23 TB = 92 TB, data stored twice = 56.8 TB. One node down leaves 69 TB raw for 56.8 TB of copies (82% raw) - so the four-hour outage already ran with zero tolerance for a second failure. Two down leaves 46 TB of raw for 56.8 TB of data: the order database goes unavailable.
- 5 nodes at RF3: raw ≈ 115 TB, usable ≈ 1/3 ≈ 38 TB. Current 28.4 TB is already 74% of usable, and a node failure leaves 92 TB raw against 85.2 TB of copies (93%) - no room to re-protect, so a second failure during rebuild is effectively unrecoverable.
- 6 nodes at RF3: raw ≈ 138 TB, usable ≈ 46 TB; 28.4 TB used = 62%, and one down leaves 115 TB raw for 85.2 TB of copies (74%), which rebuilds.
Recommendation: six nodes (add two), then convert. That also leaves ~17 TB of headroom, roughly 14 months at 1.2 TB/month.
2. The team argues "we already survived a node outage, so we meet the requirement." How do you correct that, and what changes operationally once RF3 is on?
Read the worked answer
Correct the premise first. Surviving one failure at RF2 proves exactly one thing: RF2 does what RF2 promises. The auditor's wording is "any two nodes," which RF2 cannot satisfy at any utilisation level, and at 70% used the surviving three nodes were already carrying a full second copy with no rebuild headroom. Also note the requirement is cluster-wide: there is no per-VM redundancy factor, so "just protect the order DB" is not an option inside this cluster - either the cluster moves to RF3, or that workload moves to its own 5+ node cluster.
Operational changes after conversion:
- Write amplification and inter-node replication traffic rise; expect higher network utilisation on the backplane and validate with 10/25 GbE, not 1 GbE.
- The RF change triggers Curator to re-protect every extent; this runs for hours to days. Schedule it in a maintenance window, watch cluster IOPS/latency and Curator task progress, and do not combine it with firmware upgrades.
- Capacity planning uses usable/3, not usable/2 - any spreadsheet still assuming 50% must be rewritten.
- RF3 protects availability, not unlimited rebuild: after two simultaneous failures data stays online but cannot be re-protected until a node is replaced. Stock a spare node or contract for 4-hour parts.
- Run NCC health checks before and after, and confirm snapshot/protection-domain space at the remote site is also sized for the new overhead.
3. Before the change window, what verification steps would you insist on, and what could still go wrong?
Read the worked answer
Pre-change gate: NCC clean (no failed checks on disk, CVM, or metadata), all nodes same AOS version, no pending Curator or disk-balancing tasks, and no open hardware alerts on the three existing nodes - converting RF on a cluster with a flaky disk converts a disk failure into an outage.
Capacity gate: recalculate on usable numbers, not the dashboard headline, after adding nodes; include snapshots, protection-domain retention, and any planned VM builds in the next quarter. If the result exceeds ~70% of RF3 usable, do not convert yet.
Time and rollback: estimate re-protection duration from cluster throughput and current data volume (roughly data volume divided by achievable rebuild rate), and tell the business that during that window a single failure is survivable but a second one is not. RF3→RF2 is possible but is itself a data-moving operation, so treat the change as one-way for planning purposes.
What can still go wrong: the conversion saturates the backplane and slows latency-sensitive workloads; a latent disk error surfaces mid-rebuild; the auditor turns out to want site tolerance, not node tolerance - in which case two nodes in one room is the wrong answer entirely and the conversation becomes replication or a stretched cluster.