Snapshot Sprawl in a 400-VM vSphere Estate
A university has 400 VMs on six vSphere clusters, 60 TB used, protected by PowerProtect Data Manager with eight backup proxies and a DD appliance. The policy takes a daily backup at 22:00, keeps the VMware snapshot for 24 hours, and retains 14 daily and 8 weekly copies. Over three months, datastore utilization has climbed from 68% to 91%, several VMs show snapshot consolidation warnings, and the backup window has grown from 4 to 9 hours. The team recently enabled cloud tiering to an object store for backups older than 30 days. A faculty member reports that a restored VM from last week booted but the application data was 26 hours old. The network team sees backup traffic crossing a 1 GbE uplink shared with production, and the VMware team wants to disable quiescing to reduce stun time.
1. Explain why datastore utilization rose and backups slowed. Identify at least three causal factors and distinguish backup data from VMware snapshots.
Read the worked answer
The primary cause is the 24-hour VMware snapshot retention. A snapshot is not a backup; it preserves redo data on the same datastore and grows with every write. Keeping it for a day across 400 VMs, especially busy file and database VMs, consumes datastore capacity and creates consolidation work. The consolidation warnings show that some snapshots are not being removed cleanly, so orphaned delta disks accumulate. The backup window growth is separate: eight proxies may be insufficient for 400 VMs, and the 1 GbE uplink shared with production throttles hot-add or NBD traffic. Cloud tiering for backups older than 30 days does not reduce local snapshot growth or the daily backup workload. Quiescing is also relevant: if the VMware team disables it, database and file-system consistency may degrade, but that is a data-integrity trade-off, not a fix for snapshot sprawl. The restored VM with 26-hour-old application data suggests the restore selected an older restore point, or the application was not quiesced and its own logs were inconsistent, or the backup of that VM had been failing without being noticed.
2. Redesign the backup policy and VMware integration to restore the window and stop snapshot growth.
Read the worked answer
Release the VMware snapshot as soon as the backup data stream is complete, not after 24 hours; use PowerProtect Data Manager to monitor and delete snapshots, and alert on any snapshot older than the job. Keep only the backup copies on DD for 14 daily and 8 weekly, and use cloud tier only for long-term retention. Enable CBT and hot-add, place proxies per cluster, and add proxies or 10 GbE connectivity so the eight proxies are not the bottleneck. Limit concurrent snapshots per datastore and per cluster to avoid stun and datastore burst. Keep quiescing for databases and applications that need it; use application-consistent protection rather than disabling quiescing globally. Exclude swap and temporary disks. For the 26-hour-old restore, verify the selected restore point, CBT validity, and job success, and require an application-level validation after restore. Add a weekly restore test that checks data age and application consistency, not just whether the VM boots.
3. What monitoring and governance would you put in place to prevent recurrence?
Read the worked answer
Monitor snapshot age and count per VM, orphaned delta disks, datastore free space, and consolidation failures daily. Monitor backup job success, duration, and restore-point age by VM; a green job status is not enough if the restore point is stale. Track proxy utilization and network throughput on the shared uplink, and alert when the window approaches its limit. Track cloud-tier recall latency and egress, because the restored VM issue may become a cloud-restore issue later. Governance: assign ownership for snapshot cleanup to the VMware team and backup success to the backup team, with a weekly joint review. Require change control for policy changes such as snapshot retention and quiescing. Publish an RPO/RTO statement per application tier and test restores quarterly. The university should also document that cloud tiering is archival, not a substitute for local recovery, so expectations are set before an incident.