XFS data volume fails to mount after power loss
A CentOS Stream 9 server has an 8 TB XFS filesystem on a hardware RAID6 volume, mounted at /data. The UPS failed during a firmware update, and the host lost power while a large rsync was writing. On reboot the system reaches the multi-user target, but /data is not mounted. mount /dev/mapper/vg_data-lv_data /data returns 'Structure needs cleaning'. dmesg shows XFS metadata corruption and advises unmounting and running xfs_repair. xfs_repair -n /dev/mapper/vg_data-lv_data reports several AG free-space btree discrepancies and says the log contains a dirty transaction. The last successful backup finished 18 hours ago, but 2 TB of new research data has been written since. The RAID controller reports all disks online. The application that uses /data is stopped. A storage snapshot from before the outage exists but has not been tested.
1. Decide whether to repair or restore, and lay out the safe sequence. Explain the role of the dirty log and -L.
Read the worked answer
Treat the array as evidence. First stop writes and unmount any residual mount. If the storage supports it, create an LV or array snapshot before repair, because xfs_repair modifies metadata. Run xfs_metadump to a separate volume if space allows. xfs_repair -n is read-only and confirms the extent of damage; it is not a repair. The dirty log means recent metadata transactions were not replayed. A normal mount would replay the log, but mount fails. xfs_repair without -L will refuse or attempt safe recovery when the log is dirty; -L forces the log to be zeroed and discards recent metadata operations, which can lose files or produce inconsistent data. Use -L only when the log is corrupt and a backup or restore is not preferable. Given 2 TB of new data and an 18-hour-old backup, attempt repair on the snapshot first, validate, then repair the original. If repair proposes rebuilding many AGs, restore the tested snapshot instead and accept the 18-hour loss.
2. What commands and precautions are needed before and after xfs_repair?
Read the worked answer
Before repair: verify no process has the filesystem open with lsof or fuser; unmount; snapshot; run xfs_metadump; record xfs_info and the exact xfs_repair -n output. Run xfs_repair -v on the snapshot, not the original, if possible. After repair, run xfs_repair -n again to confirm clean metadata, then mount read-only with mount -o ro,norecovery for inspection; norecovery avoids replaying a log you have not validated. Check dmesg for new I/O errors. Only then mount read-write and run application consistency checks. If xfs_repair reports a fatal error or repeatedly fails, do not keep running it with different flags; restore. Never run xfs_repair on a mounted filesystem. After a successful repair, take a fresh backup before resuming the rsync.
3. What changes reduce the chance of this recurring?
Read the worked answer
Fix the power path first: configure the UPS to initiate a clean shutdown, and test it. Enable the RAID controller's write cache policy only with battery backup; otherwise leave write-through. For /data, consider nofail and a systemd timeout so a failed mount does not block multi-user, but do not let the application start without a mount check. Keep backups and test restores; an untested snapshot is not a recovery plan. Monitor XFS metadata errors via dmesg and smartd, and alert on unclean shutdowns. Schedule xfs_repair -n only during maintenance, never automatically at boot. Use noatime to reduce metadata writes, and ensure the rsync uses a resumable strategy so a repeat outage does not require a full 2 TB copy. Document that XFS has no fsck-at-boot equivalent and that repair is a manual, disruptive operation.