Multipath I/O Stall During a SAN Fabric Upgrade
One physical database server runs RHEL 9 and PostgreSQL 15 on XFS over LVM. The 4 TB LUN is presented through two Fibre Channel fabrics and two HBAs; multipathd maps it as mpatha with path_grouping_policy multibus. /etc/multipath.conf contains no_path_retry 30, fast_io_fail_tmo 5, and dev_loss_tmo 600 in the defaults section. During a fabric A switch upgrade, one path is lost. multipath -ll mpatha shows one path failed faulty running and one active ready running, but PostgreSQL sessions block for about 95 seconds and dmesg reports SCSI command timeouts. A junior admin wants to set queue_if_no_path to prevent all stalls and to raise dev_loss_tmo to 3600. The LUN has no replication; this is the only server. multipathd was last reloaded before the fast_io_fail_tmo line was added.
1. Why did one failed path cause a 95-second stall when a second path was active and ready? What would you inspect to prove the cause?
Read the worked answer
Do not assume the defaults are live. Check multipath -ll mpatha for the feature string and multipathd show maps raw format for effective fast_io_fail_tmo, dev_loss_tmo, and no_path_retry. The setup says the fast_io_fail_tmo 5 line was added but multipathd was not reloaded; existing maps can retain fast_io_fail_tmo off. In that state, commands outstanding on the failed FC path are not failed quickly, even though another path is ready, so PostgreSQL waits through SCSI timeouts. A second path protects against path loss but not against slow failure detection on in-flight I/O. queue_if_no_path is not the cause here because one path remains; it would only matter after all paths are lost.
2. Recommend configuration changes and explain why you would not simply set queue_if_no_path and dev_loss_tmo 3600.
Read the worked answer
Apply and verify rather than just edit. Set fast_io_fail_tmo 5, set dev_loss_tmo to a value greater than fast_io_fail but not 3600, for example 30 to 60 seconds based on array behavior, and choose no_path_retry deliberately. I would not enable indefinite queue_if_no_path on a single server: it can hang PostgreSQL forever when all paths are gone. A bounded no_path_retry returns errors so the application or operator can act. Run multipathd reconfigure and confirm the live map shows the new values; some changes require a map reload or reboot, so schedule it. Test by disabling one HBA or zoning one path in a maintenance window and measure failover latency. Raising dev_loss_tmo to 3600 would make the server wait far too long for a fabric that may not recover.
3. Because this is a single-node service, what monitoring and operational safeguards would you add around multipath?
Read the worked answer
Add monitoring for path state and effective timeouts, not just LUN presence. Alert on failed or faulty paths from multipathd show paths, SCSI timeouts in dmesg, and PostgreSQL I/O latency. Keep the two fabrics on separate switches and avoid changing both at once. Document the tested failover time as the recovery time objective. Because this is the only server, pair multipath with tested backups and a restore procedure; multipath is not a substitute for node redundancy. After any multipath.conf change, verify multipath -ll and perform a controlled path-failure test. Also confirm that the storage array and HBA firmware timeouts are compatible with the host settings, so the host does not remove a path before the array has had a chance to fail over.