Idempotent VM Provisioning Against Prism Central v3 APIs
Contoso's platform team uses a Python script against Prism Central v3 APIs to build 80 lab VMs every night. The script creates a VM, attaches a disk, powers it on, and tags it. During a Prism Central upgrade, the script received HTTP 429 and 503 responses. Its retry logic re-sent create requests and produced 37 duplicate VMs, and 12 VMs were left powered off because dependent calls ran before tasks completed. The production environment uses Terraform, but the lab uses raw API calls. The CI pipeline runs four workers in parallel, and the lab must be ready by 06:00. A previous attempt to serialize everything made the run exceed the window.
1. Design an idempotent provisioning flow for the raw API script. What must change in create, retry, and cleanup?
Read the worked answer
The script currently treats a POST as safe to retry. It is not. Idempotency has to be designed in.
- Deterministic naming and external keys. Derive VM name, disk name, and a lab-run identifier from the run ID and role (for example, lab-0421-web-01). Before creating, query by name and by a category or annotation that stores the run ID. If the object exists and matches the desired spec, skip creation; if it exists but is wrong, update or delete and recreate under a controlled path.
- Create-then-verify. Treat a timeout as an unknown outcome, not a failure. After a timeout or 5xx, query for the object by name before retrying the POST. Only retry the create if the object is absent.
- Retry policy. Retry only idempotent GETs freely. For POSTs, retry only after the existence check. Use exponential backoff with jitter, cap attempts, and honor Retry-After on 429. Do not retry 4xx validation errors.
- Dependency ordering. A VM create returns a task UUID. Poll the task until success before attaching a disk or powering on. Do not fire dependent calls in parallel with the create task. Model the workflow as a state machine: create VM -> task complete -> attach disk -> task complete -> power on -> task complete -> tag.
- Concurrency. Four workers are fine only if they operate on disjoint VM sets and share a rate limiter. A global semaphore or queue prevents a thundering herd during Prism Central recovery.
- Cleanup. At the end of the run, reconcile desired versus actual: delete orphans tagged with the run ID that were never completed, and report VMs that failed validation. A separate dry-run mode should list intended actions before applying them.
2. How should the script handle async tasks, 429 responses, and Prism Central upgrades without missing the 06:00 deadline?
Read the worked answer
Treat Prism Central as an eventually consistent control plane, not a database.
Async tasks. Every mutating call that returns a task UUID must be polled to a terminal state with backoff. Do not assume a 202 means the VM is usable. Poll with increasing intervals, set a per-task timeout, and distinguish task failure from task timeout. On timeout, query the resource state rather than re-issuing the mutation. Log the task UUID for every step so the run can be resumed.
429 handling. A 429 is flow control, not an error to hammer through. Honor Retry-After, add jitter, and reduce worker concurrency dynamically. Put a token-bucket limiter in front of all API calls. If 429s persist, the correct response is to slow down, not to add retries that amplify load.
Upgrade windows. Prism Central upgrades can cause 503s and temporary API unavailability. The pipeline should detect a maintenance signal, pause new work, and resume from the last completed step using the run ID. Schedule lab builds outside Prism Central upgrade windows where possible, and add a preflight health check that fails fast if the API is unhealthy.
Meeting 06:00. The 80-VM run has a fixed deadline, so measure the critical path. Parallelize independent VMs, but serialize steps within a VM. Use a persistent run journal so a restart resumes rather than recreates. If the window is still too tight, pre-create a pool of stopped VMs and power them on at run time, or reduce per-VM steps. Serializing the entire run was the wrong fix; bounded concurrency with idempotency is the right one.
3. Should the lab move to Terraform as well? Compare it with the raw API approach and state the guardrails.
Read the worked answer
Terraform is a better default for declarative, repeatable infrastructure, but it is not automatically safer for an 80-VM nightly lab.
Terraform advantages. It tracks state, plans changes before applying, and handles dependency ordering. It reduces duplicate creation because state maps resources to real objects. It gives a reviewable diff and a clean destroy path. For production, that is the right control surface.
Terraform costs. Nightly create/destroy of 80 VMs can produce large state churn. State must be stored remotely with locking, or four parallel workers will corrupt it. Drift and partial applies still need reconciliation. Importing pre-existing lab VMs is possible but tedious. If the lab is ephemeral and identical every night, Terraform works well; if it is highly dynamic, the raw API may be simpler.
Recommendation. Use Terraform for the lab's baseline topology and the raw API only for runtime operations Terraform does not model well. Do not run both against the same objects without ownership rules.
Guardrails for either approach:
- Remote state with locking and one writer per workspace.
- A naming and tagging standard that encodes environment, run ID, and owner.
- A plan/approval step for production; automated plan-only for lab.
- Rate limiting and bounded concurrency in the provider or wrapper.
- A reconciliation job that finds untagged or orphaned VMs and reports them before deleting.
- A tested rollback: destroy by run ID, not by broad name pattern.
- Audit logging of who or what created each VM, tied to a service account rather than a shared admin.