Stalled Rebuild and Slow I/O After a Disk Replacement
A four-node cluster running AOS with RF2 had a disk failure on node 3. The drive was replaced and the cluster recognized the new disk, but Curator's rebuild has sat at 38% for six hours. Prism Element shows Data Resiliency: Degraded and cluster storage at 84% used. VMs on node 4 are reporting high I/O latency, and an NCC run fails one check related to cluster services. The on-call engineer wants to reboot all three CVMs to clear the state and, if that does not work, remove node 3 from the cluster.
1. Give your first three diagnostic steps, in order, and justify the ordering.
Read the worked answer
Confirm the current failure surface before changing anything: check CVM health on all nodes, and look at the alerts page for disk, CVM, and resiliency alerts with timestamps. You need to know whether this is one bad disk or a node-level problem.
Check free capacity. At 84% used with RF2, the cluster may simply not have room to rebuild efficiently; rebuild competes with live I/O for the same spindles and the same free extents. This also tells you whether removing node 3 - the engineer's fallback - would make things dramatically worse by reducing capacity further.
Check whether rebuild is stalled or just slow: look at Curator and Stargate activity in the CVM logs on node 3 and node 4, and check whether a second disk on the same node is degraded or whether the replaced disk is healthy. A silently failing second disk is a common reason a rebuild appears to freeze.
Ordering matters because steps 1-2 are read-only and rule out the two situations where the engineer's plan is actively dangerous.
2. Why is rebooting the CVMs the wrong first move, and what should be done instead?
Read the worked answer
Rebooting all three CVMs removes the cluster's ability to serve I/O and coordinate rebuild at the moment it is already degraded: if any replica is temporarily unavailable, VMs can stall or the cluster can lose quorum behavior it needs. It also destroys the in-memory and log evidence that would explain the stall. Two of the three CVMs rebooting simultaneously on a four-node RF2 cluster is not a maintenance action, it is an outage.
Instead: identify which specific service is unhealthy - one CVM, one disk, or Curator on one node - and address that. If a single CVM is genuinely unresponsive and the cluster is otherwise healthy, restarting that one CVM is reasonable. Before any restart, capture the logs, because after the restart the evidence is gone.
Removing node 3 should be off the table entirely: with 84% used and a degraded rebuild, removing a node reduces capacity and failure tolerance at the worst possible time.
3. What do you collect for support, and what interim actions reduce user impact?
Read the worked answer
Collect a log bundle from Prism Element (or the logbay equivalent on your version) covering the period from the disk failure through now, plus the NCC output and its failing check name, the alert list with timestamps, current capacity by container and RF, and the replacement disk's health and SMART data. Include the rebuild progress over time as evidence that it is stalled rather than merely slow. Note the AOS version and the exact disk model, since replacement-drive compatibility is a frequent culprit.
Interim actions: move the highest-latency VMs off node 4 while rebuild pressure continues, if the cluster will migrate them; pause non-essential snapshot and replication schedules that add write load; and stop any manual disk or node operations until support has reviewed the bundle. Provide support with a single timeline and a named owner rather than a stream of separate updates.
Also open the conversation about capacity: 84% used on an RF2 cluster is the underlying reason this incident is painful, and it will not fix itself when the rebuild completes.