The Storage Array That Outgrew Its Snapshot Policy
A four-node virtualization cluster presents 120 TB of VMFS datastores from a single all-flash array. The array runs 15-minute snapshots retained for 72 hours, plus daily snapshots retained for 30 days, on every datastore. Over six weeks, write latency on the datastores climbs from 1.2 ms to 9 ms, and the array's snapshot pool reaches 84% of its 40 TB capacity. Array CPU sits at 55%. The vendor's guidance states that snapshot pool consumption above 80% triggers automatic snapshot deletion, oldest first. The backup team reports that nightly jobs now exceed their 6-hour window, finishing in 8 hours 20 minutes. Application owners have not reported errors yet. The storage team proposes buying 20 TB of additional snapshot pool capacity.
1. Diagnose why latency and backup duration degraded. Should the team approve the capacity purchase as the primary fix?
Read the worked answer
The symptoms point to snapshot overhead, not raw array throughput. Array CPU at 55% and a snapshot pool at 84% of 40 TB indicate that copy-on-write metadata and pool pressure, not front-end saturation, are driving the 1.2 ms to 9 ms latency increase. Every 15-minute snapshot on 120 TB of datastores creates a large, churning set of delta extents; as the pool fills, the array has less room to place new writes efficiently, and garbage collection competes with guest I/O. The 8h20m backup time is consistent with the same cause: reads during backup traverse snapshot metadata, and a nearly full pool slows that traversal.
Buying 20 TB treats the symptom. It restores headroom but leaves the retention design that generated 84% consumption, so the team will likely be back at the threshold within a similar six-week window. Worse, the 80% auto-deletion policy means the current state is one busy week away from silently losing recovery points that the business believes it has.
A defensible response: (1) immediately verify which snapshots would be auto-deleted and whether any are the only copy of required recovery points; (2) reduce snapshot frequency on datastores that do not need 15-minute RPO, or consolidate to a smaller set of high-value datastores; (3) investigate whether backup software is mounting snapshots and leaving them mounted, which extends delta growth; (4) only then consider capacity, sized to the redesigned policy rather than the current one. Treat the purchase as a buffer during remediation, not the remediation.
2. The business requires a 15-minute RPO for a subset of tier-1 VMs. How should the team reconcile that requirement with the array's snapshot behavior?
Read the worked answer
The team should separate the RPO requirement from the mechanism used to meet it. A 15-minute RPO does not require 15-minute array snapshots on all 120 TB; it requires that tier-1 VMs can be recovered to a point no older than 15 minutes. Options include:
- Scope 15-minute snapshots to only the datastores holding tier-1 VMs, and move those VMs to dedicated datastores so the policy is enforceable and measurable.
- Use application- or guest-level replication for tier-1 workloads, which can meet a 15-minute RPO without multiplying array-level delta extents across every datastore.
- Keep array snapshots but lower retention on the high-frequency tier (for example, 15-minute snapshots retained 4 hours rather than 72), since the RPO governs recovery point, not retention depth.
The team must also confirm that the 80% auto-deletion threshold cannot violate the RPO. If the array can delete the snapshots that satisfy a contractual RPO without alerting anyone, the control is not real. Set alerting well below the threshold, and document the deletion behavior as a known risk with an owner.
Finally, validate the RPO by test restore, not by policy review. A 15-minute snapshot cadence that takes 40 minutes to mount and restore does not deliver a 15-minute recovery capability in practice.