RAID6 rebuild stalls on a 12-disk array
An Ubuntu 22.04 backup server uses a 12-disk mdadm RAID6 array of 8 TB SATA drives as the sole backing store for LVM thin pool vgbackup. One drive failed and the hot spare began rebuilding. Two days later /proc/mdstat still shows the rebuild at 4 MB/s with an estimated 30 days remaining. Application write latency has risen from 5 ms to 250 ms, and some backup jobs are timing out. dmesg shows repeated link resets on the replacement drive, which was taken from cold spare stock and is the same model as the failed units. mdadm --detail shows a write-intent bitmap is present. The array is not the boot device. The team can tolerate slower backups but not data loss, and the next full backup starts in 12 hours.
1. Diagnose why the rebuild is slow and decide whether to change rebuild speed, replace the new disk, or both. What evidence do you need?
Read the worked answer
4 MB/s on 8 TB drives is not a tuning problem; it is two orders of magnitude below normal. Check cat /proc/mdstat, mdadm --detail /dev/md0, dmesg | grep -i -E 'ata|reset|link', smartctl -a on the replacement, and iostat -x 5. Repeated link resets and a cold spare from the same batch point to a failing or unsuitable drive. Also check whether the model is SMR; SMR drives are catastrophic for RAID rebuilds. Read /sys/block/md0/md/sync_speed_min, sync_speed_max, and sync_action. Raising sync_speed_max may improve throughput only if the disk is healthy; it will not fix resets and may increase latency. Because the array is RAID6 and already degraded, replace the suspect spare rather than trusting it. Do not use --assume-clean.
2. What is the safe procedure to replace the suspect spare without losing redundancy, and how do you protect the thin pool?
Read the worked answer
Protect redundancy first. Confirm the array state and that a second hot spare is available or add one before touching the rebuilding disk. If the replacement is the problem, run mdadm --fail /dev/sdX, mdadm --remove /dev/sdX, then mdadm --add /dev/md0 /dev/sdY with a known-good CMR disk. If the disk is still in the array and you want to replace it in place, mdadm --replace is an option, but failing a suspect spare is simpler. Do not remove the original failed disk's slot until the new member is fully synchronized. Set a moderate sync_speed_max during backup hours to protect latency, then raise it after the backup. Keep the write-intent bitmap; it prevents a full rebuild after an interrupted resync. After rebuild, run mdadm --detail and check mismatch_cnt, then validate the thin pool with lvs, thin_check, and a test restore. Take a fresh backup before any further maintenance.
3. What monitoring and configuration changes prevent a 30-day rebuild from going unnoticed?
Read the worked answer
Use mdadm --monitor --scan --mail or a monitoring agent that parses /proc/mdstat and /sys/block/md0/md/sync_completed. Alert on degraded state, spare count below one, resync speed below a floor, and ETA above a threshold. Add SMART monitoring for all members and for cold spares; burn in spares before adding them. Track thin pool data and metadata usage separately, because a full metadata LV can make the pool read-only. Alert on dmesg ATA resets and on mismatch_cnt changes. Document the rebuild procedure and test it in a lab. Keep at least one validated spare of the same capacity and CMR technology. Review RAID6 versus RAID10 for backup workloads if rebuild time is the dominant risk. Finally, do not treat a hot spare as a backup: the 12-hour full backup still needs a separate copy.