/var is full but du says 61 GB - the deleted-inode decision
app-prod-07 runs RHEL 9 with /var as a 500 GB ext4 filesystem on an LVM logical volume. At 02:14 the on-call engineer receives an alert: filesystem /var at 100 percent. df -h /var reports 500G size, 500G used, 0 available; df -i /var reports 3 percent inode use; du -shx /var totals 61 GB. Applications are logging 'No space left on device' and the audit daemon has stopped writing. lvs shows the volume group has zero free extents, so extending requires a new LUN and a storage ticket with a four-hour SLA. The duty manager suggests rebooting the host to clear the problem. A colleague notices that PID 2214, the log shipper, has been running since before the last maintenance window, and that three weeks ago the logrotate stanza for /var/log/app/*.log was changed from copytruncate to create. A change freeze is in effect until 06:00.
1. df and du disagree by roughly 440 GB. Explain what accounts for that gap, identify the most likely cause, and list the checks you would run to rule out the alternatives before making any change.
Read the worked answer
Start with the arithmetic. 500 GB is allocated to the filesystem and only 61 GB is reachable as files, so about 439 GB is held by something du cannot see. ext4 reserves 5 percent of blocks for root by default, so roughly 25 GB of the 500 GB is reserved-block overhead (tune2fs -l /var | grep -i reserved). That leaves approximately 414 GB genuinely unaccounted for.
The explanation that fits every symptom is a deleted-but-still-open file. Three weeks ago logrotate was switched from copytruncate to create. The new mode renames the live log, creates a fresh one, and expects the writer to reopen it. PID 2214 was never restarted and still holds the file descriptor to the renamed, then deleted, inode. The inode and its data blocks stay allocated until the last fd closes, which is why df sees them and du does not.
Confirm it with lsof +L1, lsof -p 2214, and ls -l /proc/2214/fd, which will show a path ending in (deleted).
Rule out, in this order:
- Inode exhaustion - already excluded,
df -ishows 3 percent. - Reserved blocks - only 25 GB, tune2fs output confirms.
- LVM snapshots or a thin pool holding deleted data -
lvs -aanddmsetup status. - A filesystem mounted over a directory in /var, hiding data -
findmnt -R /var. - Sparse versus real allocation -
dureports blocks, so this would not produce a 414 GB gap.
Do not reboot yet: it would clear the symptom without a root cause and destroy the state you need.
2. State the immediate remediation that restores service inside the change freeze, and justify it against rebooting or killing PID 2214. What risks does your chosen action leave behind?
Read the worked answer
Free the blocks without touching the process: truncate -s 0 /proc/2214/fd/<n> where <n> is the descriptor ls -l /proc/2214/fd shows pointing at the deleted file. The kernel releases the data blocks immediately, the shipper keeps its descriptor, and no live log content is lost because the file is already unlinked.
Why not the alternatives:
- Reboot clears the inode but gives you no root cause, may not be approved during a freeze, and takes down a production host - disproportionate for a disk-space incident.
kill -9on PID 2214 frees the inode too, but can drop buffered events, and if the service is not supervised it may not come back. It is a bigger change than needed at 02:14.
Residual risks to state explicitly:
- The shipper continues writing at its previous offset, so the file becomes sparse:
ls -lshows a large apparent size whiledustays small. This is expected, not a new leak, but it confuses the next responder. - Space will grow again at the same rate because the writer still has no reopen mechanism. This is a stopgap measured in hours.
- Data written to /var and to the audit log during the ENOSPC window was lost; auditd stopped by design when it could not write. After freeing space,
systemctl restart auditdand verify withauditctl -sthat it is enabled and logging.
Schedule the real fix - restarting the shipper so it opens the current file - for the Sunday window, and confirm afterwards that its fd points to an existing path.
3. Design the permanent fix and the monitoring that would have caught this before the filesystem hit 100 percent.
Read the worked answer
Permanent fix, in the logrotate stanza. Keep create (it avoids the line loss that copytruncate causes) but add a postrotate block that tells the writer to reopen: systemctl reload log-shipper or a HUP to the process. copytruncate is not automatically the right answer - it can drop records written between the copy and the truncate, and it competes with the writer. Choose the mode based on whether the shipper supports reopen, and if it does not, restart it from postrotate with sharedscripts so it happens once per rotation, not once per file. Add maxsize so rotation also triggers on volume, not only on schedule, and su plus explicit create 0640 modes so ownership is predictable.
Capacity. Set an 80 percent threshold on both space and inodes and act at it, while the volume group still has extents. Pipe lvs free-extent counts into monitoring so a full VG is an alert, not a discovery. Review whether the default 5 percent reserve is appropriate for /var; tune2fs -m 1 returns about 20 GB here, but only after the team accepts that non-root processes can then fill the disk completely.
Detection that would have fired weeks earlier. Alert on total size of deleted-but-open files per host - a small collector running lsof +L1 and exposing the sum as a node_exporter textfile is enough - plus an alert on auditd not running, on any application logging ENOSPC, and on disk growth rate (GB per hour) projected to reach 100 percent within 24 hours. The original 02:14 alert was the last possible moment; every one of these fires days before it.