Storage Spaces Direct Cluster Losing Capacity
Tailspin has a four-node Storage Spaces Direct cluster, each node with 12 x 4 TB NL-SAS drives and 2 x 1.6 TB NVMe drives used for cache. The cluster hosts 60 CSV volumes and about 400 VMs. Resilience is three-way mirror. Usable capacity was planned at roughly 40 percent of raw.
Over three months, free space on the pool falls from 45 percent to 8 percent with no new VMs deployed. Administrators report that volume optimization jobs fail with insufficient free space, and that the cluster occasionally pauses a node for maintenance. A backup product creates a nightly snapshot of every volume and keeps 30 days. Event logs show repeated disk warnings on node 3.
A proposal is to add four more 4 TB drives per node and rebuild the pool to recover capacity.
1. Identify the most likely causes of the capacity loss and rank them by likelihood, with the evidence you would collect for each.
Read the worked answer
1. Snapshot accumulation (most likely). A nightly snapshot per volume with 30-day retention means up to 30 snapshots per volume, and each snapshot holds changed blocks. With 60 volumes and 400 VMs, churn is substantial. Snapshots consume pool capacity invisibly to volume free-space reporting. Evidence: Get-ChildItem on the volume shadow copy store, vssadmin list shadows, and the pool's free space compared with the sum of volume free space. A large gap between the two confirms snapshot consumption. Also check whether the backup product deletes snapshots reliably - a failed job can orphan them.
2. Disk warnings on node 3 (likely). Repeated disk warnings suggest drives are being retired or are intermittently failing. A retired drive forces the cluster to rebuild its copy elsewhere, which consumes capacity for the rebuild and, if rebuild cannot complete, leaves volumes in a degraded or reduced-resilience state. Evidence: Get-PhysicalDisk | Where OperationalStatus -ne OK, Get-StoragePool health, and the Failover Cluster / Storage Spaces Direct event channels for disk retirement events.
3. Volume optimization not completing (contributing). Optimization reclaims space from deleted blocks. If it fails for lack of free space, deleted data is not reclaimed and the pool fills further - a feedback loop. Evidence: job history and the error text from the optimization task.
4. Thin provisioning overcommit (possible). If volumes were thinly provisioned beyond pool capacity, the 40 percent planning figure may have been optimistic. Evidence: compare provisioned size to pool capacity.
Least likely: an unexplained metadata leak. Rule out the above first.
2. Evaluate the proposal to add drives and rebuild the pool, and recommend an alternative sequence.
Read the worked answer
Evaluation. Adding four 4 TB drives per node adds 64 TB raw. At three-way mirror that is roughly 21 TB usable, which buys time but does not address the cause. Rebuilding the pool is worse: it requires evacuating or destroying volumes, is a multi-day operation at this scale, and if the underlying cause is snapshot retention, the new pool will fill at the same rate. Rebuilding also discards the cache configuration and any tiering, and it is a high-risk change on a cluster already pausing nodes for maintenance.
Recommended sequence.
- Stabilize. Resolve the node 3 disk warnings first. Replace failing drives and let rebuild complete before any capacity change. A cluster in a degraded state should not be expanded.
- Reclaim. Delete orphaned snapshots and fix the backup product's retention so snapshots are removed after the backup completes rather than held for 30 days. This is usually the largest single recovery. Then run volume optimization with enough free space to succeed.
- Measure. Re-check pool free space against the sum of volume free space. If the gap closes, the cause was snapshots and no hardware is needed.
- Expand only if still needed. Add capacity by adding drives to existing nodes and letting the pool expand - this is an online operation and does not require a rebuild. Do not rebuild the pool.
- Right-size resilience. If the workload tolerates it, consider a mix of three-way mirror and parity volumes rather than three-way mirror for all 60 volumes; this can materially change usable capacity.
Judgement. The proposal treats a symptom. The exam-relevant reasoning is to separate capacity consumed by data from capacity consumed by snapshots and rebuilds, and to avoid a destructive rebuild before the cause is known.