Online pvmove from a failing SAN LUN on a full volume group
A two-node Oracle Linux 8 database VM keeps /u01 on an XFS filesystem on logical volume u01lv in volume group vg_u01. The VG is built from two 500 GB multipath LUNs, /dev/mapper/mpatha and /dev/mapper/mpathb, each on a separate array. /u01 is 91% full and the VG reports 0 free extents. The storage team says the primary path to mpatha is degrading: dmesg shows repeated 'I/O error, dev dm-3' and 'Buffer I/O error on dev dm-3', a few times per hour, and the array will be replaced in 48 hours. The application can be stopped for no more than 10 minutes. A third 1 TB LUN, mpathc, is zoned but multipath -ll lists it with a single path and no alias; it has never been partitioned. The DBA proposes: 'Just pvcreate the new disk, vgextend vg_u01, and run pvmove /dev/mapper/mpatha - it is online, so no downtime.'
1. Evaluate the DBA's plan. Why can pvmove fail even though the new 1 TB LUN is available, and what must be true before the move starts?
Read the worked answer
Assessment. pvmove does not move data to unallocated space outside the VG; it allocates free extents in the same VG as the temporary mirror destination. vg_u01 currently has 0 free extents, so a pvmove attempt would fail with an insufficient-free-space error before protecting anything. The new LUN must first become a PV in vg_u01 with vgextend, and it must present as a single multipath device. multipath -ll showing one path and no alias is a warning: either the second path is not zoned or visible, or multipathd is not managing it. Adding a partially visible LUN as a PV can create duplicate PVs and corruption. Check multipath -ll, lsblk, pvs, and dmesg; fix zoning or multipath.conf, run multipath -r, and confirm one wwid. Also confirm the used extents on mpatha are less than the free extents added, take an XFS metadata backup, and watch for read errors because pvmove must read the failing extents. If mpatha has unreadable blocks, pvmove can stall or fail and restore from backup is the fallback.
2. Give an ordered command sequence to migrate the extents off mpatha with /u01 mounted, and list the checks that prove it is safe to remove the old LUN.
Read the worked answer
Sequence. Confirm the application is idle enough and take a backup or storage snapshot. Confirm multipath with multipath -ll and dmsetup ls --tree. Create the PV and extend: pvcreate /dev/mapper/mpathc, vgextend vg_u01 /dev/mapper/mpathc, pvs -o+pv_name,pv_size,pv_free,pv_used, vgs. Start the move with progress: pvmove -i 10 /dev/mapper/mpatha. While it runs, lvs -a -o+devices shows the temporary mirror and dmesg -w watches for new I/O errors. When it completes, pvdisplay /dev/mapper/mpatha should show 0 used and lvs -o+devices vg_u01/u01lv should show only mpathc. Then vgreduce vg_u01 /dev/mapper/mpatha and pvremove /dev/mapper/mpatha. Verify xfs_info /u01, mount | grep u01, and application read/write. Only then ask storage to unpresent the LUN. The move is online; the 10-minute outage is only needed if pvmove hits I/O errors and you must fail over to backup. Keeping the old LUN until verification gives a rollback path.
3. How should /u01's 91% usage be handled in the same change, and why is the order of pvmove and lvextend important?
Read the worked answer
Capacity and order. pvmove needs free destination extents equal to the used extents on mpatha. If you first extend u01lv to consume all of mpathc, you use the free space that pvmove needs and the migration fails. So migrate first, remove the failing PV, then extend. With XFS mounted, use lvextend -r -l +100%FREE vg_u01/u01lv; the -r calls xfs_growfs on the mounted filesystem, which is the supported grow path for XFS because there is no resize2fs. If you must extend before pvmove, leave headroom and target the new PV explicitly, for example lvextend -l +50%PVS vg_u01/u01lv /dev/mapper/mpathc, but that complicates allocation and is unnecessary here. After growing, confirm df -h /u01, xfs_info /u01, and that LVM did not allocate new extents on mpatha. Also set monitoring below 91% and investigate the growth source. XFS growth is online but cannot shrink, so if the 1 TB LUN is temporary, plan permanent capacity rather than treating it as a one-off.