Thin Pool at 96% While the Guest Says 55%

Guest df reports allocation inside the virtual disk. It says nothing about physical blocks in the thin pool.

Each snapshot creates a copy-on-write exception store. The first write to a chunk of the origin after a snapshot exists forces LVM to preserve the pre-write contents, and those blocks are charged to the pool, not to the origin's filesystem. With 41 snapshots and a database doing continuous random writes across 5 TB, the oldest snapshots have accumulated weeks of churn - 41 overlapping exception stores, all growing independently.

That is why the numbers diverge: the guest sees a 5 TB filesystem with 2.75 TB of live data; the pool sees that plus every preserved pre-image block. data_percent at 96% with no autoextend means you are one burst of writes from a pool-full event that makes every thin volume in the pool read-only or failing writes.

The 1 TB of free extents is only about 10% headroom at the current growth rate, so it buys time, not safety. Treat this as a capacity incident with a root cause in the backup job, not as a filesystem problem.

Do not add space first. If the runaway snapshots are still being created, new space is simply consumed.

1. Stop the producer. Disable the snapshot timer or cron entry so no new snapshot is created mid-operation. This is reversible and takes seconds.
2. Extend the pool, online. lvextend -L +700G vg/thinpool, and extend metadata proportionally (lvextend --poolmetadatasize +8G vg/thinpool). Metadata at 68% is also approaching the point where the pool can go read-only, and metadata exhaustion is far harder to repair than data exhaustion. Leave a reserve for metadata; it cannot be shrunk later. No downtime is required.
3. Remove snapshots oldest-first. lvremove one at a time, reading lvs -o name,data_percent after each. Oldest snapshots hold the most exception blocks, so the first removals give the largest recovery and the clearest signal that you are freeing space. Keep the newest snapshot if a backup is mid-window.
4. Enable autoextend as a guard rail. thin_pool_autoextend_threshold = 80, autoextend_percent = 20 in lvm.conf, then vgcfgbackup. This is a safety net, not a substitute for fixing the script.
5. Fix the script. set -euo pipefail, a retention query that matches by origin rather than a path glob, and a pre-flight refusal when data_percent > 80. Add an alert on the job's exit status.

All of this is online; the two-minute database stop is not needed for any step.

Guest df monitors a virtual number that cannot fail on its own; the failure lives one layer down, where thin provisioning decouples virtual size from physical blocks.

Signals worth alerting on:

• lvs -o data_percent,metadata_percent - warn at 70/65, page at 85/75. Metadata gets the tighter threshold because a full metadata LV makes the pool unusable and needs lvconvert --repair.
• Snapshot inventory per origin - count and maximum age. A rule such as 'snapshots older than retention exist' fires the day the deletion branch breaks, which is the actual root cause here.
• VG free extents and a 7-day growth projection - days-to-full is more actionable than a percentage.
• dmsetup status vg-thinpool - the same percentages read from the kernel side, useful when LVM tooling is slow or wedged during an incident.
• Backup job exit status and duration - the silent failure was a job failure; monitor the job as a first-class target.

Correlate with host-level write throughput so the alert says 'the pool is growing because of 41 snapshots on a busy origin', not just 'the pool is growing'.