The 38 GB That du Cannot See

The df/du gap is the signature of unlinked inodes that still have open file descriptors. du walks directory entries; an unlinked file has no entry, so du cannot see it, but the kernel keeps the blocks allocated until the last fd closes.

Confirm before acting:

lsof +L1 | grep -i deleted      # or: lsof -nP +D /var | grep deleted
df -h /var

Expect one ~38 GB entry owned by PID 3421, probably /var/log/app/app.log (deleted).

Truncate the descriptor instead of restarting the service:

: > /proc/3421/fd/7        # N = the fd number from the lsof NAME column
df -h /var                 # blocks are released immediately

Trade-offs you should state explicitly:

• Restarting app.service is cleaner but drops in-flight sessions and needs the same approval you are trying to avoid; truncation keeps the process alive.
• The JVM keeps writing at its old file offset, so the file becomes sparse and ls -l still shows a large size. df is the source of truth.
• If the fd points at a socket or pipe rather than a regular file, the truncate fails with EINVAL and a restart is the only option.

Do not delete or move anything under /var/log first - that is what created the situation.

Use the one-step online grow:

lvextend -r -L +10G /dev/vg0/var

-r invokes fsadm, which selects xfs_growfs for XFS and resize2fs for ext4, so you do not need a separate grow step and no unmount is required.

Constraints:

• XFS can only grow, never shrink. Extending by the full 20 GB is irreversible without a rebuild, so take 10 GB now and keep headroom in the VG for snapshots.
• If vg0 were a thin pool, check lvs -a -o name,data_percent,metadata_percent first; thin volumes can be extended beyond pool free space and fail later at write time.
• Verify with df -h /var and lvs afterwards rather than trusting the exit code alone.

Most importantly, say out loud that the extension buys time but does not fix the cause: without a rotation fix, a 60 GB volume will hit the same wall in roughly two months at the current rate.

The failure mode is that logrotate rotated the file but the JVM kept writing to the old inode. Two acceptable fixes, in preference order:

1. Reopen-capable rotation - if the application exposes a reopen endpoint or signal:

/var/log/app/*.log {
  daily
  rotate 14
  compress
  missingok
  notifempty
  sharedscripts
  postrotate
    systemctl reload app.service
  endscript
}

This is the correct fix because the writer is told to reopen the new path.

1. copytruncate - for a JVM with no reopen hook:

  daily
  rotate 14
  compress
  copytruncate

Accept and document the race: lines written between the copy and the truncate are lost.

Detection, layered:

• An alert at 85% used on /var, not 100%.
• A check that greps lsof +L1 output for regular files over 1 GB and alerts on the PID and size - this catches the failure mode directly, regardless of filesystem usage.
• logrotate -d /etc/logrotate.d/app in CI or after any unit change to confirm the config still parses and matches the expected path.