Reclaiming a Six-Layer qcow2 Chain on a 92%-Full Datastore
An internal file server VM, filesrv, uses a qcow2 disk vda backed by a chain of six images on a 1.2 TiB NFS datastore: a base image of 500 GiB virtual and 500 GiB allocated, plus five external snapshots created monthly, each 40-180 GiB allocated. The datastore is 92% full, with about 96 GiB free. The VM must remain online; the SLA allows an I/O pause of at most 60 seconds but no reboot. Nightly backups currently take 11 hours because the backup job reads the whole chain. qemu-img info --backing-chain shows the active layer is the sixth image. You have a four-hour maintenance window tonight and can attach one additional 2 TiB NFS datastore that is already mounted on the host. Monitoring reports write-latency spikes whenever the nightly backup overlaps business hours.
1. Why is a single live `blockcommit --active --pivot` unsafe on this datastore, and how much free space does it need?
Read the worked answer
virsh blockcommit filesrv vda --active --pivot merges every backing image downward into the base file and then pivots the running guest onto that base. The base grows by the allocated data held in the layers being committed - here roughly 550 GiB, since the chain totals about 1.05 TiB and the base is 500 GiB - plus qcow2 L2 metadata and copy-on-write overhead while the job runs. Those clusters are allocated in the base before the upper layers are deleted, so the peak requirement is current usage plus the committed data, not the net result. With only 96 GiB free, the job hits ENOSPC. If the NFS filesystem is genuinely full, guest writes to the active layer can also fail, which is an availability incident, not just a failed maintenance job.
Calculate the requirement rather than guessing: qemu-img info --backing-chain /path/filesrv.vda gives each layer's allocated size, and qemu-img measure can estimate the destination file size. Require the sum of allocated layers plus at least 10% headroom. qemu-img commit offline is not a loophole: it needs the VM down and the same space. The correct conclusion is that this datastore cannot absorb a full-chain commit.
2. Give a safe procedure that fits the four-hour window and the 60-second I/O pause limit. Include the commands and the order of operations.
Read the worked answer
Take the copy path first, because it does not grow the nearly full datastore.
- Save the definition and confirm the current disk:
virsh dumpxml filesrv > filesrv.xml,virsh domblklist filesrv --details,qemu-img info --backing-chain /images/filesrv.vda. - Copy the whole allocated chain to a single qcow2 on the new 2 TiB datastore with blockcopy:
virsh blockcopy filesrv vda /mnt/newstore/filesrv-new.qcow2 \
--format qcow2 --wait --verbose --pivot
Blockcopy keeps the guest running and only pivots at the end, so the I/O pause is brief. At roughly 300 MB/s, 1.05 TiB takes about an hour, well inside four hours.
3. Verify the pivot with virsh domblklist filesrv --details and qemu-img info /mnt/newstore/filesrv-new.qcow2. Only then delete the old chain.
If the new datastore is unavailable, commit the smallest top layers into their parents first:
virsh blockcommit filesrv vda \
--base /images/filesrv.snap4 --top /images/filesrv.snap5 \
--wait --verbose
The peak extra space equals the allocated size of the top layer (about 40 GiB), which fits in 96 GiB. Delete-and-repeat works for the 60 GiB layer, but the 120 GiB and 180 GiB layers exceed free space, so stop and use blockcopy for those. Never run --active --pivot across the whole chain on this datastore.
3. How would you verify the result and prevent the chain from growing again?
Read the worked answer
Immediately after the operation, confirm there is one image and that the guest is using it: qemu-img info --backing-chain should show no backing file, virsh domblklist --details should point at the new path, df -h should show real free space, and qemu-img check should report no corruption. Watch guest write latency and NFS retransmits for the rest of the window; a blockcopy that completed but left the NFS server saturated is still a failed change.
Prevention is a policy change, not a one-off cleanup. Cap the chain at two external snapshots and consolidate weekly rather than monthly. Alert at 75% datastore usage, not 92%, and require a qemu-img measure check before creating a snapshot. Take backups from a dedicated backup datastore using a pull or export, not by reading the live chain for 11 hours. If snapshots are used for rollback, create them with virsh snapshot-create-as --disk-only --atomic --quiesce and delete them after the backup completes. Finally, document the free-space rule in the runbook: any commit or pull operation needs the allocated size of the layers it will write, plus headroom.