Thin pool exhaustion with a rollback snapshot
An Ubuntu 22.04 PostgreSQL host uses LVM thin provisioning. VG vgdata contains a 3.6 TB thin pool thinpool and two thin volumes: pgdata 2.8 TB formatted XFS and mounted at /var/lib/pgsql, and pgsnap 1.1 TB, a snapshot taken before a migration. vgdata has 25 GB free. lvs reports the pool data at 97.8% and metadata at 41%; df -h /var/lib/pgsql shows only 61% used. Monitoring watches df only, so no thin-pool alert fired. The snapshot is 11 days old and must remain available for rollback until migration sign-off in 48 hours. You have one spare 2 TB LUN that can be attached online. Writes now fail intermittently with 'No space left on device' even though XFS has free blocks. The database cannot be taken offline during business hours.
1. Explain why df and lvs disagree and why XFS writes fail despite free blocks.
Read the worked answer
XFS sees the size of the pgdata thin volume, not the free space in the thin pool that backs it. df reports 61% used because the filesystem still has unallocated blocks within its 2.8 TB volume. lvs reports the pool at 97.8% because thin volumes and the snapshot consume pool chunks on write; the snapshot's copy-on-write and PostgreSQL's writes have nearly exhausted the shared pool. When the pool fills, the thin target cannot allocate a new chunk, so writes to pgdata fail even though XFS believes it has free space. That is why the error is intermittent: some writes reuse already-allocated blocks, while new allocations fail.
Confirm with lvs -o name,size,data_percent,metadata_percent,pool_lv vgdata, dmsetup status vgdata-thinpool, and dmesg for thin-pool errors. Do not run xfs_growfs or delete files as the first response; the bottleneck is pool allocation, not XFS size.
2. What is the safest immediate remediation using the spare LUN while keeping the snapshot and database online?
Read the worked answer
Attach the 2 TB LUN, verify the device name with lsblk and multipath -ll if applicable, then run pvcreate /dev/sdX, vgextend vgdata /dev/sdX, and extend the thin pool, for example lvextend -L +1.8T vgdata/thinpool. This gives the pool new chunks and should stop the ENOSPC failures without growing or touching pgdata. Record lvs before and after. Because the pool is nearly full, avoid operations that force large allocations, such as creating another snapshot or running a merge now.
After extension, verify data_percent drops and test a small PostgreSQL write. Do not remove pgsnap until the migration is signed off; it is the rollback artifact. If the pool still fills quickly, identify the consumer with lvs and dmsetup status; the snapshot may be consuming most of the new space. Schedule any snapshot merge for a maintenance window, because merging can be I/O-heavy and is not a same-minute fix.
3. What monitoring and LVM policy changes would prevent a repeat after the 48-hour rollback window?
Read the worked answer
Monitor data_percent and metadata_percent, not just df. Export the values from lvs --noheadings -o data_percent,metadata_percent vgdata/thinpool into the monitoring agent and alert at 75% and 85%, with a critical alert at 90%. Also alert on dmsetup status thin-pool state and on dmesg thin-pool errors. A filesystem-only check would have missed this incident entirely.
Enable LVM autoextend only after adding free VG capacity: set thin_pool_autoextend_threshold and thin_pool_autoextend_percent in lvm.conf and ensure dmeventd is running. Autoextend cannot help if the VG has no free extents, so keep headroom in vgdata. Adopt a snapshot lifetime policy: snapshots must have an owner, an expiry, and a rollback window; after sign-off, remove pgsnap promptly. Test the alert path by lowering a threshold in a lab or by simulating the metric, not by filling the production pool.