LFCS: A 900 GB log directory with 2 million files and a daemon holding deleted inodes

A 900 GB log directory with 2 million files and a daemon holding deleted inodes

On a RHEL 9 host, /var/log/app holds roughly 2 million files spread across 400 subdirectories written by a vendor daemon. du -sh /var/log/app reports 900 GB and the volume is full. ls /var/log/app times out. The vendor states that the daemon keeps file descriptors open on rotated files and will continue writing into deleted inodes until it is restarted; restarting drops in-flight jobs, which you must avoid during business hours. Only coreutils, findutils and util-linux tools are available - no third-party utilities. File names follow app-*.log within each subdirectory, but subdirectories also contain other file types you must not touch.

1. Describe how you would attribute the growth to specific subdirectories without ever running a bare `ls` on the top level.

Read the worked answer

Run a bounded walk rather than a directory listing:

cd /var/log/app
ionice -c3 nice -n19 du -x --max-depth=1 --block-size=1 . | sort -n | tail -20

Reasoning: du reads directory entries and stats them - it never needs the sorted, fully-materialised output that ls builds, so a huge directory does not blow up memory the way ls can. -x keeps it on one filesystem so a nested mount is not charged to the wrong subtree. --max-depth=1 bounds the work to 400 subtrees; sort -n on raw bytes avoids -h's rounded comparison.

Expect minutes, not seconds: ionice -c3 and nice keep it off the critical path, and you should run it in a tmux session rather than an SSH one-liner that dies with the connection.

To go one level deeper on a suspect subtree, repeat with --max-depth=1 scoped to that path, and cross-check the file count with find <subdir> -maxdepth 1 -type f -printf '.' | wc -c (which streams, unlike ls).

2. Write the find-based cleanup that removes `app-*.log` files last modified more than 14 days ago, and explain the two ways this command can destroy more than you intended.

Read the worked answer

Dry run first, always:

find /var/log/app -type f -name 'app-*.log' -mtime +14 -print | wc -l
find /var/log/app -type f -name 'app-*.log' -mtime +14 -print | head -20

Then delete:

find /var/log/app -type f -name 'app-*.log' -mtime +14 -delete

-delete handles the file list internally, so it does not hit ARG_MAX the way xargs rm can on 2 million paths; if you do pipe, use -print0 | xargs -0 -r rm -f.

Two ways to over-delete:

-mtime rounding. -mtime +14 means age rounded down to whole days is strictly greater than 14, i.e. files that are 15 or more days old. If the requirement is "older than 14 days" literally, -mtime +13 is the closer match for files 14 days and 1 second old. State which semantics you implemented; do not guess.
A bad start path or a missing -type f. If /var/log/app is a symlink or a mount point that shifts, or if the run happens from the wrong CWD, find still happily deletes. Add -xdev to stay on one filesystem, keep -name quoted so the shell does not glob it, and re-read the -print output before removing the -print.

3. After the deletion, `df -h` shows almost no space recovered and the daemon is still running. Explain what is happening, how you prove it, and what you would do instead of restarting the daemon.

Read the worked answer

The files are unlinked but not freed: the daemon still holds open descriptors, so the inodes and blocks persist until it closes them. df counts blocks in use, not directory entries, so the space does not appear.

Prove it:

lsof +L1 /var/log/app 2>/dev/null | head
du -sh /var/log/app   # much smaller than before
lsof -p <daemon_pid> | grep deleted | head

If du shrank but df did not, deleted-but-open files are the explanation - no ambiguity.

Instead of restarting, reclaim space by truncating through the descriptor without closing it:

: > /proc/<pid>/fd/<n>
truncate -s 0 /proc/<pid>/fd/<n>

Both keep the same inode and offset semantics the daemon expects; the process continues writing at its current offset, so the file can regrow as a sparse file but the space is released now. Do this per descriptor, largest first, and re-check df after each.

The durable fix is upstream of all of this: move the backup/log target off the exhausted volume, add a capacity alarm on df --output=pcent, and get the vendor to reopen files on rotation (SIGHUP) so the next cleanup actually returns space.