df Says Full, du Says 12 GB
A Linux web server alerts at 03:12 that /var, a 50 GB volume, is 100% full. The on-call admin runs du -xsh /var/* and totals about 12 GB, while df -h /var still reports 50G used with 0 available. Nightly at 02:00 a cron job rotates the nginx logs: it renames access.log and error.log aside, compresses the copies, then deletes the uncompressed originals. Nothing signals nginx afterward. The nginx master process has been running for 41 days. lsof is installed. Meanwhile the application returns 502 errors because it cannot write session files under /var/lib/app/sessions. The admin's proposal is to reboot the host.
1. Reconcile the 50 GB in df with the 12 GB in du. Name the mechanism and the command you would use to confirm it.
Read the worked answer
du walks directory entries; df counts allocated blocks on the filesystem. A file that has been unlinked has no directory entry, so du cannot see it, but its blocks stay allocated until the last open file descriptor is closed. The rotation script renamed the live logs and then deleted the only names, while the nginx worker processes - running since before the rotation, for 41 days - still hold those inodes open and keep writing to them. Roughly 38 GB is sitting behind deleted file descriptors.
Confirm with lsof +L1 (or lsof /var | grep deleted, or ls -l /proc/<nginx-master-pid>/fd), which lists open files with link count 0 and their sizes. The sizes should sum to approximately the df−du gap. Note du -x also excludes mounted filesystems, but df and du are reporting on the same volume here, so that is not the explanation.
2. Identify the two mistakes in the rotation job and give the immediate remedy, including its main risk.
Read the worked answer
Mistake 1: the job never tells nginx to reopen its log files. A rename-based rotation must be followed by nginx -s reopen (or kill -USR1 <master-pid>) so the workers close the old inode and open the new path. Note that kill -HUP is a config reload, not a log reopen - a common source of this exact bug.
Mistake 2: the job deletes the uncompressed name while a writer still holds the file open. That removes every trace and defers the space accounting to process exit, which is why the gap appears hours later rather than at 02:00.
Immediate remedy: truncate the still-open descriptors, e.g. : > /proc/<pid>/fd/<n> for each deleted log, which frees the blocks instantly and leaves the writer with a valid (now empty) file. Do not rm - there is nothing left to remove - and do not rely on > access.log in the rotated path, which touches a different inode.
Risk: truncation destroys whatever log content is still in the file. If those logs matter for an ongoing investigation, copy them out first via /proc/<pid>/fd/<n> before truncating.
3. Explain the 502 errors and evaluate the reboot proposal.
Read the worked answer
The 502s are a downstream symptom, not a second fault: with zero free blocks the application cannot create session files, its upstream handler fails, and nginx returns 502 to clients. Fixing space fixes the 502s.
A reboot probably "works" because terminating nginx closes the file descriptors and releases the 38 GB, but it is the wrong first move:
- It destroys volatile evidence (process state, /proc fd data, in-memory logs) that you may want if the log deletion turns out to be relevant to an incident.
- It extends the outage well beyond the seconds needed to truncate or signal nginx.
- It guarantees recurrence: the cron job is unchanged, so at the next 02:00 rotation the volume fills again, likely at a worse time.
Correct order: free space (reopen/truncate), confirm with df, verify the app writes sessions and 502s stop, then fix the rotation script to reopen logs after rotation, and add monitoring that alerts on the df-versus-du gap or on lsof +L1 output so a deleted-open file is caught the night it happens.