Clone Capacity Wall in a Shared Dev Cluster
A platform team uses one NDB deployment to manage production and development. Production runs a 1.2 TB PostgreSQL 14 database on a three-node AHV cluster; its time machine takes a snapshot every 4 hours and retains snapshots for 30 days. The development cluster has three nodes and 20 TB raw capacity, with 5 TB currently free. Twelve application developers each request a clone of production for a two-week feature sprint. They expect to modify about 20 GB per clone per day and want a daily clone refresh. NDB creates the first eight clones successfully, but the ninth fails with a storage-capacity error. Prism Element shows 5 TB free before the clones, and the NDB clone job log points to insufficient space on the development container. The developers also plan to refresh clones every morning after the previous evening's production snapshot. One clone already has a local snapshot retained for testing.
1. Why can thin clones exhaust a cluster even when the base snapshot is shared? Estimate the storage pressure and identify the additional consumers.
Read the worked answer
Thin clones are space-efficient, not zero-cost. The base snapshot may be shared, but each clone has its own VM disks, changed blocks, logs, temp files, and NDB metadata. Twelve clones changing 20 GB per day for 14 days is about 3.36 TB of unique changed data before overhead. Add database logs, temp space, VM swap, and snapshot metadata, and 5 TB of free capacity is marginal.
There are additional consumers that are easy to miss. A daily refresh can temporarily require space for both the old delta and the new delta, and it may create an intermediate snapshot. If the clone is placed on a different cluster from the source, NDB may need to copy or stage the base data, which can consume a full-size copy rather than a thin reference. The retained local snapshot on one clone also pins blocks and prevents reclamation.
The ninth-clone failure is therefore a capacity-threshold symptom, not proof that only eight clones are possible. Check Prism container free space, AHV reserve, NDB metadata growth, and the size of clone deltas after several days. The estimate should include a safety margin; do not plan to consume 100% of free capacity.
2. Propose a clone strategy and refresh policy that meets the sprint without risking the dev cluster. What must be checked before the daily refresh?
Read the worked answer
Reduce concurrent clone count or add capacity. A practical plan is to create clones in waves, set a 14-day expiration, and stagger refreshes so not all twelve clones churn at once. Use fewer long-lived clones with shared application schemas where possible, and delete the retained local snapshot unless it is required for a specific test. Monitor per-clone growth and enforce a change budget; 20 GB per day per clone should be treated as a planning assumption, not a guarantee.
For refresh, confirm the source production snapshot is complete and registered in the time machine. Confirm the clone has no retained local snapshot, dependent clone, or active NDB job that would block refresh. Quiesce the clone application if the release requires it, verify free space for the new delta plus temporary overhead, refresh during a defined window, and validate application connectivity and row counts afterward. Refresh one batch at a time and stop on the first capacity warning.
If the development cluster cannot absorb the churn, place clones on a separate cluster, shorten the sprint retention, or use a smaller production snapshot as the clone source. Do not let the daily refresh become an unreviewed capacity event.