Rebuild Stall and RF Decision on a Full 8-Node Cluster
Acme Manufacturing runs an 8-node NX-3060 cluster on AOS 6.5 with RF2 and reports 62% logical capacity used. The cluster hosts 340 VMs, including a 90 TB cold archive workload with deduplication and compression enabled, plus a busy SQL tier. Last Tuesday node 7 failed at 02:10. Curator began rebuilding, but the rebuild took 9 hours 40 minutes, and the SQL tier saw latency spikes throughout that window. The team also noticed that node 7 had been 88% full before failure while other nodes averaged 61%. Capacity growth is 4% per month. The next budget cycle can fund either two more nodes or an RF3 conversion. The CISO wants RF3 for the archive data; the platform team wants to know whether rebuild time, capacity risk, or performance is the real problem before spending.
1. Diagnose why the rebuild was slow and why the SQL tier suffered. What evidence would you collect before changing anything?
Read the worked answer
Rebuild is not a simple network copy. After a node loss, Curator scans and copies extents to restore RF2, and it competes with foreground I/O. The likely causes here are cumulative:
- Fill imbalance. Node 7 was 88% full while peers averaged 61%. After failure, the surviving seven nodes absorbed its data unevenly, and the fullest surviving node became the rebuild bottleneck. Rebuild does not magically balance across nodes.
- Cold archive with dedup/compression. Rebuilding a 90 TB deduplicated container requires reconstructing fingerprints and metadata, which is CPU- and metadata-heavy. Dedup lowers physical writes but increases work per extent during repair.
- Rebuild throttling. Curator has a default rebuild rate and may be limited by disk throughput on the busiest node rather than by network bandwidth.
- Foreground contention. Losing one of eight nodes removes 12.5% of read/write capacity while rebuild traffic is added, so SQL latency rises even if the rebuild is correctly throttled.
Evidence to collect: per-node capacity and disk latency before and during the event; Curator and rebuild logs; extent distribution and any flagged disks on node 7; dedup savings ratio for the archive container; rebuild rate and scan settings; AHV/Prism performance charts for the SQL VMs (CPU ready, storage latency, IOPS); network utilization; and whether node 7 had prior disk warnings. Without that, adding nodes or converting to RF3 treats symptoms and may not fix the bottleneck.
2. Compare adding two nodes against converting to RF3. Which do you recommend, and what capacity reasoning supports it?
Read the worked answer
RF3 is a cluster-wide durability setting, not a per-container switch, and it changes usable capacity from roughly one-half of raw to roughly one-third. Moving from RF2 to RF3 therefore cuts usable capacity by about one-third at the same raw footprint. At 62% used today, an immediate RF3 conversion would likely push the cluster past safe operating headroom, especially with 4% monthly growth and a rebuild reserve requirement. RF3 also requires enough fault domains and is typically a five-node-minimum design; converting without capacity headroom increases the chance of a full-cluster I/O freeze.
Adding two nodes increases raw capacity by 25% and adds rebuild parallelism, but it does not fix imbalance if data is not rebalanced. The recommendation should be staged:
- Rebalance and free space on the fullest nodes first, because the 88%/61% skew is the immediate risk.
- Add the two nodes and let Curator redistribute; verify per-node utilization is within a reasonable band.
- Re-evaluate RF3 only after usable capacity after conversion still leaves adequate reserve for N+1 rebuild and 6-9 months of growth.
- If the CISO requires RF3 specifically for archive data, consider isolating archive on a separate cluster or object storage rather than forcing cluster-wide RF3 on the SQL tier.
The decision rule: RF3 is justified only when post-conversion free space can absorb a node rebuild plus projected growth. Otherwise two nodes now, RF3 later.
3. Give an ordered 60-day remediation plan that addresses rebuild exposure, performance, and capacity.
Read the worked answer
Order matters because several fixes conflict.
Days 0-7: stop the bleeding. Rebalance the cluster and remediate the fullest node. Check disk health on all nodes. Record baseline latency and rebuild settings. Do not convert RF3.
Days 7-21: reduce rebuild blast radius. Move or tier the 90 TB cold archive to a separate workload domain or object storage if policy allows. Lower dedup scope or disable dedup for archive if the savings do not justify rebuild cost. Tune Curator rebuild rate only after measuring disk latency; increasing it blindly worsens SQL latency. Set alerts for node fill above 75% and for rebuild duration beyond a threshold.
Days 21-45: add capacity. Procure and commission two nodes. Validate AOS version compatibility, run LCM prechecks, and expand the cluster during a change window. Confirm data redistribution completes and per-node utilization is balanced.
Days 45-60: reassess durability. Recompute usable capacity under RF3 using actual post-rebalance numbers, including rebuild reserve and 4% monthly growth. If RF3 fits, plan it as a separate approved change with a rollback and performance test. If not, document the risk acceptance and set a capacity trigger for the next expansion. Also test a controlled node failure in a lab or during a low-risk window to validate rebuild time and alerting.