Expanding a Three-Replica FusionStorage Pool During Rack Maintenance
A FusionStorage 6.x block pool has six nodes in two racks, ten 4 TB SATA disks per node, and three replicas. Raw capacity is 240 TB, usable is about 80 TB, and 74 TB is allocated, so the pool reports 92 percent used. The customer has approved a 90-minute power-off of rack B for UPS maintenance and separately wants more capacity. Their plan is to add three nodes to rack B, change the pool from three replicas to two to gain space, and run the rack power-off while the new nodes are rebalancing. They argue that two replicas doubles usable capacity and that rebalancing is a background job, so the maintenance window will not matter.
1. Critique the two-replica proposal from a durability and failure-domain perspective.
Read the worked answer
Two replicas would increase usable capacity, but it halves the number of tolerated failures and changes the failure model. With three replicas the pool can lose two copies and still serve data; with two replicas it can lose one. At 92 percent used, rebuild space is already tight, so a single disk failure during rebuild creates a real risk of data unavailability. The two-rack layout matters as much as the replica count: a rack-level event can remove multiple nodes at once. If replicas are placed across nodes without rack awareness, losing rack B may take two of the three copies for some extents, which is exactly the scenario the maintenance window creates. The team should confirm the pool's failure-domain setting before making any change; if rack awareness is not configured, the three-replica pool is less resilient than the customer believes, and two replicas would make a bad situation worse. Capacity should be solved by adding nodes, not by reducing protection.
2. Is it safe to power off rack B during rebalancing? Explain the interaction between rebuild, rebalance, and the maintenance window.
Read the worked answer
No. Rebalancing after adding nodes is a data movement operation that consumes disk and network bandwidth and makes placement temporarily uneven. Powering off rack B at the same time removes live capacity and forces reconstruction work on top of an already loaded pool. If the pool detects missing copies during rebalance, it may start rebuilding, and rebuild competes with client IO, extending the maintenance window and raising latency for running VMs. At 92 percent utilization there is little slack for temporary extra copies. The safer sequence is to expand and rebalance first, verify health and free space, then perform the rack power-off as a separate change with the pool balanced and monitored. If the UPS work cannot wait, postpone the expansion; the two changes have different risk profiles and should not share a window.
3. Give a step-by-step expansion plan and the acceptance checks before the rack maintenance is allowed.
Read the worked answer
Step one, confirm current health, free space, and failure-domain configuration; record per-node disk and network utilization. Step two, add nodes one at a time, waiting for the pool to return to healthy and for rebalance to settle before adding the next; adding three at once multiplies migration load. Step three, after expansion, verify usable capacity and that allocation stays below the agreed threshold, for example 75 percent, so rebuild headroom exists. Step four, test a controlled single-disk failure in a maintenance window or review rebuild rate and duration from logs, not just the dashboard. Step five, run the rack power-off only after the pool is balanced, health is green, no rebuild is active, and monitoring is live. Acceptance checks: no degraded extents, rebalance complete, latency within service level, free capacity sufficient to rebuild a full node, and a rollback or delay criterion agreed in advance. Reject the two-replica change unless a documented risk acceptance exists.