/var Reports Full While du Says It Is Not
At 03:05 monitoring pages: /var on an AIX 7.2 application server is at 100 percent. It is a JFS2 filesystem, 40 GB, on a dedicated logical volume, with no snapshot configured. df -g shows 40 GB used and 0 free, but du -sg /var reports only 12 GB. The largest files are ordinary: wtmp was rotated last week and the compressed archives total 800 MB, and there are no core dumps. The application is a long-running Java process writing rollovers into /var/app/logs; the rollover script renames the current log, gzips the renamed file, and then deletes the original. The last application restart was 41 days ago. The service is business-critical and the change board will not approve a restart before 20:00. datavg has 400 GB free. The on-call administrator proposes running fsck on /var to reclaim the missing space.
1. What is the most likely explanation for the df/du gap, and how do you confirm it while the system stays up?
Read the worked answer
The gap between df and du means allocated blocks exist that no directory entry points at. The classic cause is a deleted file that is still open: the rollover script renames the log, compresses it, deletes the uncompressed file, but the Java process still holds the original inode open and keeps writing to it. The blocks stay allocated until the last file descriptor closes, so df sees them and du cannot, because there is no path left to walk. The 41-day uptime fits: the writer has been running since before the rollover that unlinked the file.
Confirm without a restart: lsof +L1 lists open files with a link count of zero, and filtering by /var shows the holder, its PID and its size. fuser -dV /var (where available on your level) also reports processes holding deleted files. Cross-check the arithmetic: if the deleted file is roughly 28 GB, the 40 GB used versus 12 GB visible adds up exactly. Also check lsfs /var to rule out a snapshot and mount to rule out a filesystem mounted over a directory inside /var.
fsck is the wrong move twice over: it cannot run on a mounted filesystem, and it would not reclaim an open-but-unlinked file in any case.
2. What do you do at 03:15 given that a restart is forbidden until 20:00, and how do you return the filesystem to its intended size afterwards?
Read the worked answer
Availability first, root cause second. Grow the filesystem online with chfs -a size=+10G /var, which JFS2 supports while mounted, and confirm with df -g. Ten GB buys roughly a day at the observed growth rate and, more importantly, keeps the writer from failing mid-transaction. Growing is also the reversible direction: nothing needs to be undone to be safe at 20:00.
In parallel, get the owner of the Java process committed to a restart at 20:00 and have the rollover script fixed in the same change: either stop truncating in place and re-open the file, or send the process a signal so it reopens its log after rotation (copytruncate-style handling). Verify after the restart that df and du converge; until the process exits, the 28 GB will not come back regardless of what else you do. Do not try to free it by killing a thread or by deleting more files; the space is held by the file descriptor, not by a path.
After the restart, reclaim the headroom during a maintenance window with the filesystem quiet, since shrinking JFS2 is not an operation to perform under production load. Then decide whether /var should carry permanent extra capacity and add the lsof +L1 check to the weekly health script.
3. If `lsof` is not installed and the deleted-file theory does not hold, what else explains blocks allocated with no visible file on a JFS2 filesystem, and how do you check?
Read the worked answer
Work outward from the filesystem to the volume. First re-verify the mount point: mount and df /var confirm you are looking at the LV you think you are, because a stale NFS mount or an overlay mount can make df report a different device than expected. Check lsfs /var for a snapshot attribute and check whether an external JFS2 snapshot filesystem built from /var exists and is mounted somewhere else; the snapshot file consumes space in its own filesystem, but a snapshot of /var held open can also affect how space is reclaimed.
Next check for a process whose working directory was deleted, which pins the directory's blocks in the same way an open file does; fuser -cu /var plus a directory listing of /proc/<pid>/cwd identifies it. Then look at the LV itself: lslv shows whether the LV is larger than the filesystem, which by itself is normal, but a filesystem that was never grown after an LV extension can produce confusing numbers. istat on the filesystem's log and on a few known files confirms the block accounting.
If all of that is clean, a corrupt filesystem is the remaining possibility, and that is a scheduled-outage decision with a verified backup, not an in-window fsck on a live production mount.