A Failed Fibre Path and a Missing LUN
An AIX 7.2 LPAR has two FC adapters, fcs0 and fcs1, each zoned through a separate SAN switch to both controllers of a mid-range array. datavg contains four LUNs. After a switch SFP replacement, errpt shows repeated 'PATH HAS FAILED' entries for hdisk5, and lspath shows the fscsi0 path to hdisk5 as failed while the fscsi1 path is enabled. A second LUN, hdisk6, no longer appears at all: it is absent from lspath output, and lsdev -Cc disk lists it as Defined rather than Available. Applications are still running, although batch throughput on datavg has dropped by about 40 percent. The storage team says the array-side mapping was not changed. A junior administrator suggests rmdev -dl hdisk5, a cfgmgr, and then deleting hdisk6 as well so both can be 'rediscovered clean'.
1. How do you confirm that hdisk5's problem is path-level rather than a failing LUN, and what is wrong with deleting the device?
Read the worked answer
Start with the path layer, not the disk layer. lspath -l hdisk5 shows one path failed and one enabled, which already tells you the LUN is reachable and serving I/O; a dead LUN would normally fail every path. Then lsmpio -l hdisk5 (or the array vendor's path utility) shows per-path error counts and the last error, and errpt -a on one of the PATH HAS FAILED entries gives the fscsi parent and often the WWPN, which you match against the switch port the storage team just touched. If errors are climbing on fscsi0 only and the fscsi1 path counter is clean, the fault is between the adapter and the array port, not the LUN.
Deleting the device is the wrong move for two reasons. It discards the path and reservation state that chpath needs to bring the path back, and on a disk that belongs to a VG it risks leaving the VG metadata and the device definition out of step, which is a much bigger recovery than a path enable.
The right sequence is to confirm the physical fault is cleared, run cfgmgr -v to rediscover, and if the path remains in failed state, enable it with chpath -l hdisk5 -p fscsi0 -s enable (or rmpath then mkpath if the path definition itself is stale).
2. hdisk6 is Defined with no paths. What are the plausible causes, and why is 'delete it so it gets rediscovered' dangerous here?
Read the worked answer
Defined means the ODM knows a device by that name but there is no active path to it. Candidates: both paths were lost and AIX removed the paths but kept the definition; the LUN was re-presented with a different LUN ID so cfgmgr would create a new hdisk rather than revive this one; or zoning for that specific target port is still incomplete after the SFP work, so the adapter never sees it.
It is dangerous because hdisk6 is a member of datavg. Deleting the definition of a PV that still carries VGDA copies and live logical volume partitions can leave the VG with missing disks and no way to reconcile them, and if a new hdisk is later created with the same LUN the PVID mismatch will confuse the import. The safe check is metadata first: lspv to see whether hdisk6 still has a PVID and which VG it belongs to, lspath and lsdev -Cc adapter for the fscsi state, and array-side verification that the LUN is still mapped to this host's WWPNs. The storage team's statement that mapping is unchanged is worth testing, not accepting: a LUN ID change after re-zoning is exactly the kind of change that is invisible to them.
If the LUN truly is gone, that is a data-loss incident for datavg and needs a restore plan, not a cfgmgr.
3. After the path is restored, what do you verify, and what would you change to shorten the next recovery?
Read the worked answer
Verify in layers. Path layer: lspath shows every path enabled for all four datavg LUNs, and lsmpio error counters are no longer incrementing. Device layer: lsdev -Cc disk shows all LUNs Available and lspv shows each with its expected PVID and VG. Capacity layer: lsvg -p datavg, lsvg -l datavg and lslv -m on the busy LVs confirm all copies are present and in sync, running syncvg -v datavg if any stale regions exist. Performance layer: confirm batch throughput returns to baseline, because a single-path LUN that is still being used through one adapter will show up as elevated service times on that hdisk even though lspath looks healthy.
To shorten the next recovery: get the array host ports and switch ports documented per fscsi adapter so a failed path is immediately traceable to a physical port, enable path health monitoring that alerts on path count dropping below two rather than only on errpt, and confirm the disk reserve policy matches the array vendor's recommendation for this host (an active/active array normally wants no_reserve on a single-initiator LUN, and a cluster needs the clustered policy). Most of the 40 percent throughput loss came from running on one path with nobody watching the path count; an alert on path count is cheaper than the outage.