Ext4 Reports 100% Full While du Sums to 3.8 GiB
An Ubuntu 22.04 web server keeps /var on its own 20 GiB ext4 partition. Monitoring pages at 03:10 because df -h /var reports 100% used, yet du -sh /var sums to only 3.8 GiB and df -i shows IUse% at 4%. The same partition was 41% full the previous evening. Nothing was deployed, and find /var -size +200M returns nothing. The nightly logrotate run at 23:59 was edited last week: the nginx and app stanzas were changed to rotate 14, compress, create 0640 www-data adm, and the postrotate ... kill -USR1 block was deleted because it "kept printing errors". The application is queue-worker, running as deploy with a PID unchanged since the last reboot; it writes to /var/log/queue/worker.log and keeps that file open for the lifetime of the process. systemctl status shows all units active, the app is still consuming from the queue, and the disk pressure began roughly three hours after logrotate ran.
1. What single command would you run first to test the likely cause, what output confirms it, and why do df and du disagree in this situation?
Read the worked answer
Run sudo lsof -nP +L1. The +L1 filter lists open files whose link count is zero, i.e. files that have been unlinked but are still held open. Expect a line for queue-worker with a large SIZE (roughly 16 GB), a descriptor such as 3w, and a path ending in (deleted).
Why the two tools disagree: du walks directory entries, so once logrotate renames then unlinks the old worker.log, no directory entry references those blocks and du cannot see them. df reads the filesystem's block accounting, and the blocks stay allocated until the last open file descriptor is closed. The process never closed its descriptor because the signal that told it to reopen the log was removed from the logrotate config.
Confirm cheaply with ls -l /proc/<pid>/fd | grep deleted, and rule out the other classic cause by noting df -i is at 4% - this is block exhaustion, not inode exhaustion. find returning nothing is consistent, not contradictory: unlinked files have no name to find.
2. What are the safe ways to get the space back right now, and what are the trade-offs of the two durable logrotate strategies for this service?
Read the worked answer
Immediate: the descriptor belongs to a queue consumer, so systemctl restart queue-worker is the lowest-risk action - it will reopen a fresh log, release ~16 GB instantly, and unprocessed messages stay in the queue. Truncating through /proc/<pid>/fd/N (: > /proc/<pid>/fd/3) also frees the blocks but leaves the process writing to an unlinked inode, so space silently refills and the new worker.log never receives output. Do the restart first and verify with df -h /var.
Durable options:
- Signal-and-reopen (
postrotate+kill -USR1orsystemctl reload): atomic, no lost lines, but requires the app to actually implement log reopening on that signal. The deleted block was probably removed because the app ignoredUSR1; fix the app or use the correct signal rather than dropping the block. - copytruncate: copies the file then truncates it in place. Works with any process, but is not atomic - log lines written between the copy and the truncate are lost, and with
compressit is easy to misconfigure. Acceptable for low-volume logs, bad for a busy queue worker.
3. What monitoring or verification would catch the next occurrence before the partition hits 100%, and how would you validate the logrotate change itself?
Read the worked answer
Alerting: page at 85% used, but also alert on rate of change (e.g. >2 GiB/hour growth on a 20 GiB filesystem) because 41% to 100% in three hours is the real signal. Add a check that compares df used blocks against du totals; a divergence above a few hundred MB means deleted-but-open files exist, and a nightly lsof -nP +L1 that exits non-zero on any hit turns this into an explicit, named alert.
Validation: never ship a logrotate edit untested. Run logrotate -d /etc/logrotate.d/<file> to see the expanded plan, then test in staging with a dummy writer that holds the file open (exec 3>>/tmp/x.log; sleep 3600) and force a rotation with logrotate -f. Confirm the writer's output lands in the new file and that df on the test filesystem returns to its pre-test value. Finally, put the logrotate config under configuration management so a manual edit that deletes a postrotate block shows up in review.