Load Average 47 While the CPUs Are Idle
A RHEL 9 application VM has 8 vCPU and 32 GB RAM. At 14:05 its one-minute load average reached 47 while top showed 89% idle and no process above 3% CPU. API p99 latency rose from 180 ms to 9 s, affecting 200 users. ps shows 31 Java request workers in D state. vmstat 1 reports b around 30 and wa between 60 and 90. The only shared mount is nas01:/exports/appdata at /srv/appdata over TCP with hard,timeo=600,retrans=2. NAS monitoring says the array is healthy and its own latency is low, but the network team changed a firewall rule at 13:50. The application cannot be restarted during business hours. You have root, a second SSH session, and out-of-band console access. dmesg already shows nfs: server nas01 not responding, still trying. A junior admin suggests kill -9 on the D-state workers and a reboot.
1. Explain why load average is 47 when CPUs are 89% idle. Why are kill -9 and reboot risky? What evidence would you collect first?
Read the worked answer
Linux load average counts both runnable tasks and tasks in uninterruptible sleep. The 31 Java workers in D state are blocked in the kernel, most likely on NFS I/O, so they contribute to load without consuming CPU. vmstat showing a large b queue and high wa supports that; 89% idle does not mean the system is healthy, because the workers cannot run.
kill -9 cannot terminate a task stuck in uninterruptible kernel I/O until the I/O completes or fails; the signal stays pending. A reboot may hang while trying to unmount the hard NFS mount, and it can lose in-flight writes. Collect first: ps -eo pid,stat,wchan:32,comm | awk '$2 ~ /D/', cat /proc/<pid>/stack for one worker, dmesg -T | tail, mount | grep nfs, nfsstat -c, nfsstat -m, ss -tnp | grep 2049, and a short tcpdump -ni any port 2049 on the client. Confirm the firewall change time against the first D-state time.
2. What safe remediation sequence would you follow, given the application cannot be restarted and the firewall rule changed at 13:50? Include what not to do.
Read the worked answer
Contact the network team and have them restore or confirm the rule that changed at 13:50; ask for the exact before and after rule. From the client, test nc -vz nas01 2049, and if the mount is NFSv3 also rpcinfo -p nas01 and showmount -e nas01. Check existing sockets with ss -tnp dst :2049 and retransmission counters with nfsstat -c. If the firewall dropped established state, restoring the rule may let retransmits through and unblock the workers without restarting the app.
Do not reboot, do not kill -9 the D-state workers, and do not run umount -f on an active mount with in-flight writes. If restoring connectivity does not clear the D state within a short, agreed window, drain traffic at the load balancer, stop the application cleanly, then unmount and remount /srv/appdata. Use umount -l only as a last resort and document that it detaches the mount without solving outstanding I/O. Recheck nfsstat -m options before restart.
3. What changes would prevent recurrence without weakening data integrity?
Read the worked answer
Keep the hard mount for application data; changing to soft to avoid D states risks silent short writes and data corruption. Instead, add redundancy and observability: a second NAS or multipath, a synthetic NFS probe from each client, and alerts on D-state count, nfsstat -c retransmissions, and mount latency. Put firewall changes for storage networks through the same change control as the application and require a pre-change connectivity test.
Use NFSv4.1 where supported to get session trunking and better recovery, tune timeo and retrans deliberately, and document the runbook for a stuck hard mount. Add application-level timeouts and a load-balancer drain path so an operator can isolate a bad client without rebooting it. Finally, test the firewall rule change during a window, not during business hours, and keep a rollback rule ready.