I85PREP
⌘ K
LPI Linux Essentials study materials
LPI Linux Essentials study resources

ENOSPC with free blocks, exhausted inodes and a deleted open file

LPI Linux Essentials · Put your knowledge to work in context.

All case studies

ENOSPC with free blocks, exhausted inodes and a deleted open file

At 03:10 a monitoring alert fires for an Ubuntu 22.04 database host. Applications report 'No space left on device'. df -h /var shows 20G total, 13.9G used, 69% - apparently plenty free. df -i /var shows Inodes 1,310,720, IUsed 1,310,720, IFree 0, IUse 100%. du -sh /var/* sums to 6.1G, with /var/lib/php/sessions at 4.7G and 5.9 million files. A cron job running a PHP session garbage collector was disabled six weeks ago during an upgrade. Separately, lsof +L1 shows a deleted 7.8G file held open by the mysqld process, and the used space reported by df -h does not match du. The host has a 15-minute maintenance window, 2 GB RAM and no spare inodes. The team wants to know what to delete first, what will actually free inodes versus blocks, and how to prevent recurrence.

1. Explain how inodes and blocks can each cause ENOSPC, and which evidence in the setup points to each. Why is df -h alone misleading here?

Read the worked answer

ENOSPC can mean no free data blocks or no free inodes; the kernel returns the same error for both.

  • Blocks: df -h reports space. Here 13.9G of 20G is used, so blocks are not the immediate cause.
  • Inodes: df -i reports inode use. Here IUsed equals the total, 1,310,720, and IFree is 0. Every new file needs an inode even if it is empty, so creating a session file fails with ENOSPC. The 5.9 million files in /var/lib/php/sessions are the cause.
  • Deleted but open file: du counts directory entries, not blocks held by unlinked inodes. lsof +L1 shows mysqld holding a deleted 7.8G file. Those blocks cannot be reused until the file descriptor is closed, which is why df used, 13.9G, is much larger than du, 6.1G. The arithmetic 6.1 plus 7.8 equals 13.9G.

So df -h alone is misleading twice: it looks healthy relative to the inode ceiling, and its used figure includes blocks that du cannot see. You need df -i and lsof +L1.

2. The team proposes rm -rf /var/lib/php/sessions/* first. Evaluate the risks and give a safer ordered remediation plan within 15 minutes.

Read the worked answer

rm -rf /var/lib/php/sessions/* is risky in this exact state. The shell expands * to 5.9 million pathnames before rm starts; that can exceed ARG_MAX and fail with 'Argument list too long'. Even if it starts, unlinking millions of files is metadata-heavy and can saturate the disk for far longer than 15 minutes, hurting the database.

Safer order:

  1. Stop the writer: disable the PHP-FPM pool or put the application in maintenance mode so new session files stop arriving.
  2. Free blocks fast: use lsof +L1 to get the mysqld PID and file descriptor, then truncate the still-open deleted file with : > /proc/PID/fd/FD, or restart mysqld in the window. That returns 7.8G immediately without deleting live data.
  3. Free inodes in bounded batches: find /var/lib/php/sessions -type f -mtime +1 -print0 | ionice -c3 xargs -0 -n 1000 rm -f. Start with the oldest files, which are least likely to be active sessions. Watch df -i after each batch.
  4. Re-enable the writer once IFree is comfortably above zero.

Do not delete the directory itself; the application expects it to exist with the right ownership and mode.

3. What permanent configuration changes would you make, and why won't deleting files now fix the inode ceiling?

Read the worked answer

Deleting files raises IFree only until the workload creates more tiny files. The inode ceiling is fixed at mkfs time; ext4 cannot add inodes online. So prevention means either fewer files or a filesystem with more inodes.

Concrete changes:

  • Re-enable PHP session garbage collection, or replace it with systemd-tmpfiles: a configuration line such as d /var/lib/php/sessions 1733 root root 1d cleans files older than a day. Verify the mode matches the PHP pool's expectations.
  • Put /var/lib/php/sessions on its own filesystem or a size-capped tmpfs, so a runaway session count cannot exhaust the /var inode pool.
  • Alert on df -i as well as df -h; a 69%-full filesystem with 0 free inodes is an outage.
  • If session volume is legitimately large, recreate the filesystem with a higher inode ratio (mkfs.ext4 -i 8192 or -N), but that is a planned outage, not a 15-minute fix.
  • Add a tmpfiles or logrotate rule for any other directory that accumulates one file per request.