I85PREP
⌘ K
LFCS study materials
LFCS study resources

Extending /var online when the numbers on the screen are misleading

LFCS · Put your knowledge to work in context.

All case studies

Extending /var online when the numbers on the screen are misleading

A web server VM has one 40 GB disk, VG vg0 with lv_root (ext4, mounted /) and lv_var (ext4, mounted /var). df -h /var reports 1.9 GB free; vgs reports 30 GB free in vg0; df -i shows 12% inode use. A backup agent writes a 6 GB tar to /var/backups nightly at 03:00 and now fails with ENOSPC. /etc/fstab references /var as /dev/mapper/vg0-lv_var. A 2 GB classic LVM snapshot of lv_root taken two weeks ago is 88% full, and the snapshot's autoextend settings in lvm.conf are at their defaults. The vendor supports online ext4 growth but not shrinking, and the change window is 10 minutes with no reboot.

1. Before running `lvextend`, what must you verify, and why is `df -h /var` plus `vgs` not sufficient evidence that extending vg0/lv_var is the right action?

Read the worked answer

df tells you about one filesystem; vgs tells you about the volume group. Neither tells you that the filesystem behind /var is the LV you think it is.

Verify:

findmnt -T /var                        # which device actually backs /var (bind mount? nested fs?)
findmnt /var/backups /var/lib/docker 2>/dev/null   # nested mounts that would not grow with /var
lsblk -f
lvs -o lv_name,vg_name,lv_size,data_percent
vgs -o vg_name,vg_free,vg_extent_size
df -i /var                             # you already know inodes are fine, but confirm

If /var/backups is a separate mount, or /var is a bind mount from another LV, extending vg0/lv_var adds space to a filesystem the backup never touches and the 03:00 job keeps failing. Similarly, on a thin pool df on the filesystem says nothing about pool fullness - check lvs -o data_percent for the pool before allocating.

Only after findmnt -T /var shows /dev/mapper/vg0-lv_var do the numbers line up.

2. Give the command sequence for the 10-minute window, and then address the 88%-full snapshot: what is the risk, and what is the right ordering?

Read the worked answer 
lvextend -L +20G -r /dev/vg0/lv_var

-r invokes fsadm, which runs resize2fs online for ext4. ext4 grows online; no unmount, no reboot, and fstab needs no change because the mapper path is unchanged. Verify:

df -h /var
lvs -o lv_name,lv_size /dev/vg0/lv_var
dumpe2fs -h /dev/vg0/lv_var | grep -i 'block count'

The snapshot is the trap. A classic snapshot's LV size is its COW store. At 88% of 2 GB it is close to invalidation; the default snapshot_autoextend_threshold=100 and snapshot_autoextend_percent=20 mean LVM will try to grow it automatically as it approaches full - out of the same vg0 free space you are about to allocate. Order matters:

  1. lvs -o lv_name,lv_size,snap_percent vg0 to size the problem.
  2. Decide explicitly: if the snapshot is a two-week-old artifact nobody needs, lvremove vg0/lv_root_snap and recover 2 GB. If it must live, grow it first (lvextend -L +2G vg0/lv_root_snap) or keep a hard reserve, then extend lv_var.
  3. Only then lvextend -r -L +20G /dev/vg0/lv_var.

Never let autoextend race your allocation: an autoextended snapshot that consumes the free extents can make the /var extension fail mid-window, and a 100%-full COW store invalidates the snapshot silently.