/var Is Full but du Shows Much Less Space
A FreeBSD 14.1 web server reports that /var is 98% full. df -h /var shows 19 GB used of 20 GB, but du -x -h /var | sort -h accounts for only about 9 GB. The administrator deleted /var/log/httpd-access.log yesterday with rm to free space, but df did not change. The web server is still serving requests. fstat /var shows that an httpd worker process still has a large deleted file open. There is also a directory, /var/db/appcache, containing many small files that have not been accessed in over 90 days. The administrator needs to reclaim space now without killing active client connections unnecessarily, and then prevent the same problem from recurring.
1. Explain the df/du discrepancy and give the commands to identify the process holding the deleted file.
Read the worked answer
df reports blocks allocated in the filesystem, while du walks directory entries. A file that has been unlinked no longer appears in the directory tree, so du cannot see it, but its blocks remain allocated until every process that has it open closes it. The deleted httpd-access.log is therefore still consuming space through the running httpd worker.
Use fstat /var or lsof +L1 to list deleted open files and their owning PIDs. fstat -f /var can show the file and process details; lsof is available from packages and is often clearer. Cross-check with df -h /var before and after any action. Do not simply delete more files; that will not release the already-unlinked inode. Identify the PID and confirm it belongs to the web server before signaling it.
2. How do you safely reclaim the space without unnecessarily dropping active connections?
Read the worked answer
If the daemon can reopen its log on a signal, send the appropriate signal, commonly kill -HUP <pid>, and verify with fstat that the old inode is gone. If the daemon ignores HUP, a graceful restart of the web service is the safer choice: service apache24 restart or the equivalent for the installed server. That closes the deleted file and releases the blocks. Avoid kill -9 unless the process is already stuck, because it can leave children and sockets in a bad state and does not allow a graceful shutdown.
If the file path still exists and the goal is only to truncate a live log, truncate -s 0 /var/log/file preserves the inode and is safer than rm for an open log. In this case the path is already deleted, so reopening or restarting is required. After the action, confirm with df -h /var and fstat that the large deleted inode is gone.
3. What cleanup and configuration changes prevent recurrence?
Read the worked answer
First configure proper log rotation with newsyslog rather than manual rm. The entry should rotate on size or time, keep a bounded number of old files, and signal or restart the web server as appropriate. Verify with newsyslog -v and a forced rotation. Keep logs on a filesystem with enough headroom, or separate /var/log from application data if growth is unpredictable.
For /var/db/appcache, confirm ownership and application behavior before deleting anything. Use find /var/db/appcache -type f -atime +90 to list candidates, review a sample, and remove them only if the application can regenerate the cache. Add a periodic cleanup job or application cache expiry. Finally, add a disk-space check that alerts before 90%, and investigate any future df/du mismatch with fstat or lsof instead of deleting more files blindly.