Thirty Ephemeral PostgreSQL Clones for Morning CI Runs
A CI/CD team wants 30 ephemeral PostgreSQL 14 databases for parallel test runs every morning. Production runs on Nutanix Database Service (NDB)-managed PostgreSQL 14, is 6 TB, and has a two-hour SLA policy. The current process restores a pg_dump into one development VM, runs tests serially, and takes five hours. The team wants each clone isolated, refreshed from the previous night production state, and destroyed after eight hours. NDB manages one AHV cluster with 30 percent free storage. Network profiles allow only the CI VLAN to reach database VMs. Security requires that production database credentials are never placed in CI jobs.
1. Design the NDB clone workflow and assess whether 30 simultaneous full clones fit the storage and SLA constraints.
Read the worked answer
Use NDB time-machine snapshots as the clone source. The time machine captures the production database and its log stream according to the SLA policy. Create clones from the latest snapshot, not from a logical dump. NDB clones are storage-efficient at creation because they reference snapshot data, but each clone is writable and consumes additional storage as test workloads change blocks. Thirty clones of a 6 TB database are not thirty immediate 6 TB copies, but they are also not free.
Estimate divergence: if each clone writes 50 GB during an eight-hour test run, thirty clones add about 1.5 TB plus snapshot and metadata overhead. With only 30 percent free, the team must know the AHV cluster's actual free capacity, not just the percentage. Start with a pilot of five to ten clones, measure growth, and set a clone-count cap. Stagger creation to avoid a snapshot storm and to stay within the two-hour SLA. If capacity is tight, use a smaller production subset or a dedicated NDB instance for CI rather than cloning the full 6 TB.
2. How should refresh, isolation, credential handling, and cleanup be implemented so CI never touches production?
Read the worked answer
Isolation starts at provisioning. Place every clone on the CI VLAN network profile, use a clone-specific database name and port, and never register the clone in production DNS. NDB should manage separate database users for CI; generate credentials into a secrets manager or CI secret store, not into job logs. Production credentials must not be copied into clones. If NDB supports post-clone scripts, use them to create least-privilege test users, mask sensitive schemas, and disable external extensions.
Refresh should recreate clones from the newest time-machine snapshot each morning rather than patching a long-lived clone. That gives deterministic state and avoids schema drift. Destroy clones after eight hours through an automated NDB job, with a reconciliation check that catches orphans. Keep one baseline clone only if it materially reduces creation time, and refresh it from snapshot on a schedule. Log clone creation and deletion with the CI run ID so cost and capacity can be attributed.
3. The first pilot shows clone creation taking 40 minutes and several clones failing to register. What should the team investigate, and what fallback preserves the test schedule?
Read the worked answer
Forty minutes is long for snapshot-based clones and suggests a bottleneck outside the clone metadata operation. Check AHV cluster IO and CPU during creation, the snapshot size and age, network profile and IP address pool exhaustion, DNS registration conflicts, and NDB service health. Registration failures often come from duplicate names, exhausted IP pools, or the CI VLAN not having a route to the NDB control plane. Check whether the SLA policy is taking a new snapshot at the same time as the clone burst.
Stagger clone creation and cap concurrency. Pre-create a small pool of baseline clones and refresh them, rather than creating all thirty at once. If registration remains unreliable, let CI connect by IP and port from the allowed VLAN and skip DNS for ephemeral clones, while keeping inventory in NDB. If NDB capacity cannot support thirty, reduce to the number that fits and parallelize tests within each clone using separate schemas. Do not solve the problem by giving CI production credentials or by cloning outside NDB's isolation controls.