RAID 6 Rebuild Under a Double-Failure Threat
Northline Media archives 4K camera footage on a 24-bay dual-controller array. The array holds 12 x 10 TB 7200 rpm SATA drives in a single RAID 6 volume, currently 78% utilized. Ingest is 70% large sequential writes from six edit stations; the remaining 30% is random reads of project metadata and proxies. Last Tuesday drive 7 failed; the rebuild has run 41 hours and is at 63%. This morning drive 11 reported a predictive failure. The vendor's 4-hour support contract covers parts only. The team has a spare 10 TB drive on the shelf, a change freeze starting in 72 hours, and a request from finance to add 60 TB usable before quarter end. The storage admin proposes replacing the failed drive, adding six 10 TB drives to the same RAID 6 set, and letting the rebuild finish during business hours.
1. Assess the immediate risk and give an ordered action plan for the next 24 hours, including whether to replace the predictive-failure drive now.
Read the worked answer
Risk state. RAID 6 tolerates two drive failures, so the volume is currently one failure away from data loss. A predictive failure is not a failed drive, but it means another rebuild may start on short notice. The 41-hour rebuild at 63% is competing with production ingest, so any additional reconstruction I/O raises the chance that a second drive drops out.
Next 24 hours, in order:
- Capture and verify a current full backup plus a snapshot of metadata/proxies; confirm a test restore of at least one project.
- Record SMART data and array event logs for drive 11; confirm whether the predictive failure is for reallocated sectors, pending sectors, or spin retry.
- Let the current rebuild finish; do not add drives or start a manual rebuild on the degraded set. Adding members triggers a restripe and multiplies I/O.
- Keep the spare 10 TB drive on site and pre-approved for immediate replacement. Replace drive 11 only if it actually fails or if the controller recommends proactive copy; a proactive copy also consumes rebuild bandwidth, so schedule it after the first rebuild completes.
- Throttle rebuild if the array supports it only if ingest latency breaches SLA; otherwise finish faster.
- Escalate the change freeze exception for replacement, not for expansion.
2. Evaluate the admin's plan to add six 10 TB drives to the existing RAID 6 set and recommend a layout that delivers 60 TB usable without repeating the risk.
Read the worked answer
Adding six drives to the existing 12-drive RAID 6 set would create an 18-drive-wide RAID 6 and force a restripe across 180 TB raw while the set is already degraded. At 78% utilization the restripe touches most allocated stripes; rebuild/restripe times on 10 TB SATA drives are commonly measured in days, and the wider the stripe the longer the exposure. It also mixes the finance growth request with a failure event, which is poor change control.
Recommendation: leave the existing 12-drive RAID 6 group as a single unit and build a new RAID group on a new shelf. For 60 TB usable, use 12 x 10 TB in RAID 6 (10 data + 2 parity) = 100 TB usable, or 8 x 10 TB in RAID 6 = 60 TB usable if only eight bays are available. Keep the new group on separate physical disks and, if the controller supports it, separate pools. Migrate completed projects to the new group, then expand the old group only after it is healthy and the change freeze lifts. If the workload later includes metadata-heavy random I/O, consider a small RAID 10 pool for metadata rather than widening RAID 6 further.
3. Explain why RAID 5 is unacceptable for this workload and drive size, and how stripe width affects rebuild risk.
Read the worked answer
RAID 5 is unacceptable here because a rebuild reads every surviving drive in the stripe. With 10 TB SATA drives, the unrecoverable read error rate is typically 1 in 10^14 to 10^15 bits; reading 10 TB (8 x 10^13 bits) during rebuild gives a meaningful chance of hitting an unreadable sector, and RAID 5 has no second parity to recover it. RAID 6 survives one unreadable sector during rebuild, which is why it is the minimum for these drives.
Stripe width matters because rebuild time scales with the amount of data that must be read and rewritten across all members. A 12-drive RAID 6 set rebuilds by reading 11 drives; an 18-drive set reads 17 and has more surfaces and more chances for a second failure. Wider stripes also increase the cost of parity updates for the 30% random metadata workload and make restripes longer. The practical rule: keep RAID groups narrow enough that rebuild completes in hours-to-a-day, not days, and use RAID 10 where random write latency matters more than capacity.