df Says Full, du Says 41G: The fs01 Write Failures

df reports allocated blocks on the filesystem; du walks directory entries and sums visible files. A deleted file that a process still holds open has no directory entry, so du cannot see it, but the inode and its 55 GB of blocks remain allocated until the last file descriptor closes. That explains 98G in df versus 41G in du, and lsof +L1 names the holder: PID 3312 with a deleted 55 GB file.

Ext4 reserves 5% of blocks for root by default. On a 100 GB filesystem that is roughly 5 GB only root can allocate. Non-root services receive ENOSPC once the unreserved space is gone, while root-owned processes can still write into the reserve. That is why some writes fail and others do not, and it is a warning that the reserve is nearly gone too.

Rebooting would close the file descriptor and free the blocks, but it kills the in-flight builds and addresses only the symptom. The better immediate action is to free the blocks without a reboot by truncating the open file through /proc/3312/fd/<n> after identifying the correct descriptor, or by restarting the indexer in a controlled way once uploads finish. Keep the reboot as a fallback, not the first move.

1. Confirm the holder and descriptor: lsof -p 3312 | grep deleted and ls -l /proc/3312/fd. Risk: truncating the wrong descriptor damages an active artifact, so match the 55 GB size and path.

2. Decide between truncate and restart. : > /proc/3312/fd/<n> frees blocks immediately, but the process keeps its file offset; if it continues writing at that offset, the file becomes sparse and can grow again. Restarting the indexer is cleaner but fails the two active uploads. If those uploads can be retried, restart after they finish; otherwise truncate now and watch for re-growth.

3. Do not run rm -rf /var/log/*. It destroys diagnostic evidence, breaks logrotate state, and may free little if files are held open. If logs are a factor, run logrotate -f on the specific config after checking it.

4. Verify with df -h /var, df -i /var and lsof +L1 | wc -l. Confirm free space is returning and no new deleted holders appear.

5. Protect the release: tell the build owners which two jobs may need a retry, add a temporary one-minute df check with an alert at 90%, and avoid any action that restarts the runners until uploads complete. Record the free-space value before and after each step so the incident timeline is accurate.

The root cause has two parts: a process that holds a large deleted file, and a retention policy that lets /var fill. Address both.

• Fix the indexer lifecycle: it should reopen or rotate its output on a signal, or be restarted by its supervisor after a retention job. If the deleted file came from rotation, use copytruncate only as a last resort and understand its race; otherwise have the application reopen the file on SIGUSR1.
• Tighten logrotate: enable compress, reduce rotate 30 to a justified retention such as 7-14, add maxsize so logs rotate on size as well as time, and use dateext for clarity.
• Separate /var/log onto its own volume or partition, and set a quota where the platform supports it.
• Monitor df and df -i at 80/90/95%, and alert on any lsof +L1 file over a threshold; this catches the class of problem, not just this instance.

Validation: in a test VM, fill a filesystem with fallocate, confirm the alerts fire, run logrotate -d to check syntax, force a rotation and confirm compressed archives appear, then simulate an open deleted file and confirm the lsof check detects it. Document the expected free-space baseline so a future on-call engineer can tell normal from abnormal.