Filesystem Reports Full While 59 GB of Space Remains

df -h counts data blocks; df -i counts inodes, the metadata objects that represent files. On ext4 the inode table is allocated at mkfs time and never grows. Six and a half million 2 KB session files consume about 13 GB of blocks but exhaust every inode on the filesystem, so any attempt to create a new file - including a log file, a lock file, or a session - fails with ENOSPC. That is the same errno as a full disk, which is why the application reports "No space left on device" and misleads the team.

Growing a volume extends the block pool; it does not add inodes. After growing to 200 GB the filesystem would report 59 GB free in blocks and still 0 free inodes, and new file creation would keep failing. More inodes require either a different filesystem or a rebuild: mkfs.ext4 -T news or -i 8192/-N raise the inode ratio but only apply at creation, so the fix is a new filesystem plus data migration. XFS allocates inodes dynamically and would avoid this failure mode.

Standard triage must check df -h and df -i together.

1. Stop the bleeding: put the application into maintenance mode or pause the load balancer so the session writer stops creating files and log writes stop failing silently.

2. Confirm on the host: df -i /var and ls /var/lib/php/sessions | wc -l to verify the count, and check whether log files were lost.

3. Free inodes in batches. Do not use rm -rf /var/lib/php/sessions/* - the shell expands the glob before rm starts and dies with "Argument list too long". Use find /var/lib/php/sessions -type f -mtime +1 -delete, which batches internally, or rsync -a --delete /tmp/empty/ /var/lib/php/sessions/. Run it under ionice -c2 -n7 nice -n19 because unlinking millions of files generates heavy metadata I/O.

4. Watch progress with watch 'df -i /var' and stop when IUse drops below roughly 80%.

5. Create a small reserved pool so root tooling still works, then take the application out of maintenance mode and confirm 200s.

Sequencing matters: freeing inodes before restarting the app prevents the sessions from immediately refilling the filesystem.

Session data is ephemeral and should not live on the same filesystem as application logs and state. Best options: move session storage to Redis or memcached, or mount /var/lib/php/sessions as tmpfs with an explicit size= cap so it can never consume the root volume. If sessions must stay on disk, re-enable garbage collection (gc_probability=1, a suitable gc_divisor) and add an independent safety net with a systemd-tmpfiles rule, for example d /var/lib/php/sessions 1733 root root 1d, so expiry does not depend on PHP's probabilistic cleanup. The comment claiming the cloud platform handles cleanup was never true and should be deleted.

Monitoring must alert on inode utilization percentage, not only block usage; add a per-filesystem check for both, plus a check on files-per-directory growth rate.

For a rebuild, estimate inodes from peak file count: daily new files × retention days × 1.5 headroom. Use mkfs.ext4 -i 8192 or -T news to produce more inodes per byte, accepting slightly higher metadata overhead, or choose XFS with dynamic inode allocation. Validate the estimate against a week of measured find | wc -l growth before committing.