Thin Pool at 96% With Snapshots and a Deadline

df reports the filesystem inside lv_app, whose virtual size is 1.2 TB; 45% used means the filesystem has free blocks. Thin provisioning allocates physical blocks from thinpool on write. The pool is at 96% of 2 TB, so only about 80 GB remains for all 12 volumes and the snapshot's copy-on-write growth. When the pool cannot allocate a block, device-mapper returns No space left on device to the filesystem even though df looks healthy.

Prove it with lvs -a -o lv_name,lv_attr,lv_size,data_percent,metadata_percent,pool_lv,origin,devices, vgs -o vg_name,vg_size,vg_free, pvs, and dmsetup status vg_data-thinpool. Check dmesg -T for thin pool or dm errors, and confirm the mount with findmnt /srv/app. Also check metadata_percent; 18% is not critical yet, but metadata exhaustion is a separate hard failure. Do not trust df alone for thin volumes.

1. Stop new snapshot creation and the failed backup job so COW growth does not continue; tell the business that write availability is the priority. 2. Capture lvs, vgs, dmsetup status, df, and dmesg before changing anything. 3. Extend the thin pool online from the 250 GB VG free space: lvextend -L +250G vg_data/thinpool. This is online and drops pool use to about 80%, buying roughly three days at the recent 80 GB/day growth. Verify with lvs and dmsetup status.

2. Ask audit immediately whether snap_app_daily can be released. If yes, confirm its origin and that no merge is pending, then lvremove vg_data/snap_app_daily; that frees about 400 GB. If audit needs data, mount the snapshot read-only on a separate host, extract only the required files, verify them, then remove it. 5. Do not lvremove an unknown volume, do not over-extend beyond vg_free, and do not reboot; a reboot will not add pool space and may hang writes. Do not run fstrim as the emergency fix; discard can help later but is not a guaranteed immediate release. After writes are stable, schedule the new LUN and a capacity review.

First fix the cause of the last two lvcreate snapshot failures: check dmeventd and lvm logs, free pool space, and whether the snapshot name or lock collided. A nightly backup that needs a thin snapshot must reserve enough pool headroom for both the snapshot and the day's changes; otherwise the backup itself can fill the pool. Set alerts at 70% data and 70% metadata, not 90%.

Then run a restore test to an isolated host: create the snapshot, mount it read-only or use the backup stream, restore the database and files to a scratch filesystem, run consistency checks, and compare a manifest of checksums against the source. Keep the long-term audit copy outside the production thin pool; a five-day snapshot is not a backup and its 400 GB cost is exactly the risk here. Record the restore time and retention decision, and do not re-enable the schedule until a test restore passes.