I85PREP
⌘ K
VCP-DCV study materials
VCP-DCV study resources

The Backup Job That Left Fourteen Deltas Behind

VCP-DCV · Put your knowledge to work in context.

All case studies

The Backup Job That Left Fourteen Deltas Behind

A backup product snapshots each VM, backs it up, then removes the snapshot. Over a long weekend the backup server lost its repository for about 60 hours while jobs kept retrying. On Monday: the datastore holding 30 VMs is 94% full, with roughly 180 GB free of 3 TB; 11 VMs report that disk consolidation is needed; the worst VM - a 1.5 TB file server - has 14 delta disks in a chain stretching back the full 60 hours; and two other VMs still have deltas growing at about 4 GB per hour, because a job is still running against them. The team's first instinct is to select all affected VMs in the inventory and click Consolidate at once.

1. Why is consolidating all 11 VMs simultaneously a bad plan, and what is the correct order of operations?

Read the worked answer

Why the plan is risky

  1. Transient space. For thin base disks, merging a delta grows the base by the delta's contents before the delta is deleted. With 180 GB free and a 1.5 TB VM holding a 60-hour chain, consolidation can run the datastore out of space mid-merge - and a half-merged chain is far worse than an unmerged one.
  2. Array saturation. Eleven concurrent merges generate sustained random read/write across the same datastore. Every other VM on those 30 datastore-shared VMs degrades, and the merges take longer, widening the window for another failure.
  3. Two VMs are still actively snapshotting. Consolidating a VM with a live snapshot in use either fails on a lock or, worse, removes a delta the running job still depends on.

Correct order

  1. Stop and disable the backup jobs. Nothing else proceeds until the writers are quiet.
  2. Confirm no snapshot task is in progress and no delta has been modified in the last few minutes.
  3. Create headroom before merging anything: storage vMotion the largest chains to a datastore with capacity, or expand the datastore. Target well above the largest single chain size.
  4. Consolidate the small chains first, one or two at a time, watching datastore latency and free space. These validate the process cheaply.
  5. Handle the 1.5 TB VM last and separately, preferably with the VM powered off.
  6. Never delete or rename delta files by hand - that turns a recoverable chain into data loss.

2. The 1.5 TB VM's consolidation fails after 40 minutes with a lock error. What are the likely causes, and what is your next step?

Read the worked answer

Likely causes, in order of probability

  1. A live snapshot still in use. The backup job or a retry still holds a snapshot; the delta files are locked by the VMX process. Confirm by checking the VM's snapshot tree and whether any delta's modification time is recent.
  2. A stale or orphaned chain. After 60 hours of retries, the descriptor may reference a parent that was moved or removed, leaving a broken chain. Check the VM directory listing for gaps in the sequence.
  3. An external lock - an ISO mounted from a datastore, a second backup product, or a host-level operation still running against the VM.
  4. The host lost contact with the datastore mid-merge, which surfaces as a lock error rather than a path error.

Next steps

  • Read the VM's vmware.log and the host's hostd.log for the exact file that failed to lock; that names the culprit.
  • Verify no snapshot task, clone, or backup job is running against the VM.
  • Power the VM off and retry consolidation. With the VM off there is no VMX lock and the merge runs faster and with fewer failure modes. For a 14-delta chain on a 1.5 TB disk this is the pragmatic answer, not a last resort - budget the outage rather than retrying online indefinitely.
  • If it still fails with the VM off, stop and engage support with the file listing. Manual chain surgery on a production file server is a data-loss decision, not a troubleshooting step.

3. What guardrails would you put in place so the next backup outage is caught in minutes rather than after 60 hours, and how do you keep consolidation from being the recovery strategy?

Read the worked answer

Detect early

  1. Snapshot age, not just snapshot size. The failure mode here is time. Alarm on any VM whose snapshot is older than the longest expected job plus a margin - a 2-hour threshold would have fired on Saturday morning.
  2. Snapshot size per VM, so a runaway delta on the file server raises an alarm before it consumes the datastore.
  3. Datastore free space at 20% and 10%, not only at 5%. At 94% you are already in the emergency you are trying to avoid.
  4. A scheduled report of VMs requiring consolidation, and an alert when a backup job fails repeatedly - 60 hours of silent retries is a monitoring gap, not a backup gap.
  5. SIOC enabled on the datastore, with measured thresholds, so a runaway delta does not starve every other VM on the array.

Design out the dependency on consolidation

  • Cap concurrent snapshots and snapshot lifetime in the backup product; a job that cannot remove its snapshot within a defined window should abort and alert rather than retry forever.
  • Give the backup account the minimum role it needs. It must be able to create and remove its own snapshot and nothing else.
  • Where the array supports it, prefer array-based or off-host snapshot processing so the ESXi-level delta chain is short-lived.
  • Rehearse it: a quarterly test that deliberately kills a backup job and confirms the alarm fires within minutes is worth more than a runbook nobody has executed.