Second Disk Failure During RAID Reconstruction on AFF
A four-node AFF A400 cluster runs ONTAP 9.12.1. Aggregate aggr1 contains 24 x 7.68TB NVMe SSDs in RAID-DP and has two spare disks. On Tuesday one data disk fails. The replacement has not arrived. On Wednesday a second disk in the same RAID group fails while reconstruction is at 40%. Monitoring shows aggr1 is degraded and application read latency has risen from a 2 ms baseline to 18 ms. The affected volumes use space guarantee none and have 5% snapshot reserve. The application owner asks whether data is safe, whether the aggregate can survive another failure, and whether the team should immediately replace both failed disks at once to restore redundancy faster.
1. Assess the data-protection state of aggr1 and state the immediate operational priorities.
Read the worked answer
RAID-DP tolerates two simultaneous disk failures in a RAID group, so aggr1 should still be online and serving data. However, with two failed disks and reconstruction only 40% complete, the RAID group has no remaining redundancy. A third failure in that RAID group would cause data loss for the volumes on it. The aggregate being 'degraded' is not a safe state; it is a critical state.
Immediate priorities: confirm exactly which disks failed and whether a usable spare exists; check storage disk show -broken and aggregate RAID status; reduce unnecessary load on aggr1 because reconstruction is competing with production I/O; and replace one failed disk first so reconstruction can restore one parity disk. Do not pull or replace both failed disks at the same time. Wait for reconstruction to complete before replacing the second disk. Avoid aggregate relocation or other disruptive maintenance until redundancy is restored. The latency spike is consistent with reconstruction plus degraded RAID, not necessarily an application problem.
2. What should the team change in the aggregate design and spare policy to reduce this risk in the future?
Read the worked answer
With 7.68TB SSDs, RAID-DP leaves a long reconstruction window and only two-failure tolerance. For large-capacity drives, RAID-TEC is usually the better choice because it tolerates three simultaneous disk failures and reduces the risk of a second or third failure during rebuild. The trade-off is additional parity capacity, which must be planned into usable capacity.
The team should also maintain an adequate spare policy. Two spares in a 24-disk aggregate were consumed quickly; confirm spare core requirements and assign compatible spares across the cluster so a failed disk can begin reconstruction immediately. Monitor aggregate RAID status and disk health proactively, and alert on degraded aggregates, failed disks, and long reconstruction times.
Finally, validate that volume space guarantees and snapshot reserves are appropriate for the application. Thin volumes are fine, but they do not reduce RAID risk. Capacity and rebuild planning should be reviewed whenever drive size increases.