Bulk VM API Script Hits 429 and Creates Duplicates
A Python automation script provisions 300 VMs through the Prism Central v3 API. It uses 20 worker threads and posts to /vms with a name such as web-001. After about 80 VMs, the API starts returning HTTP 429. The script retries each POST immediately and does not inspect the response body. It also ignores the task UUID and assumes success when the POST returns 202. The result is duplicate VMs, some half-configured VMs, and a support case. The platform team needs to finish the remaining 220 VMs within a two-hour window while keeping the API responsive for Prism Central UI users. The environment has 3 Prism Central VMs behind a VIP. The script has no backoff or idempotency key.
1. What is causing the 429 responses and the duplicate VMs?
Read the worked answer
Reasoning: The 429 responses indicate API rate limiting caused by 20 concurrent POST workers. Immediate retries amplify the problem because they add load instead of backing off. The duplicates are a separate but related bug: a POST that actually succeeded may return slowly or lose its response, and the script retries it without checking whether the VM already exists. A 202 response means the request was accepted for asynchronous processing, not that the VM is ready or even guaranteed to be created.
Ignoring the task UUID means the script cannot distinguish success, failure, or partial configuration. It also cannot tell whether a retry is safe. The result is predictable: throttling, repeated creates, and orphaned or half-built VMs.
2. Redesign the script's request and task handling. What controls would you add?
Read the worked answer
Reasoning: Add exponential backoff with jitter and honor the Retry-After header when present. Reduce concurrency to a small, tested level rather than 20. Make creation idempotent by using deterministic names and checking for an existing VM by name or category before posting; do not rely on the POST alone. Capture the task UUID from every accepted request and poll the task endpoint with backoff until it reaches a terminal state. Treat 202 as accepted, not complete.
Record each VM's task UUID and final state in a durable log so a rerun can skip completed work. Use a bounded queue and a per-run concurrency limit. For failures, classify whether a retry is safe; do not blindly retry a create. Test with 5 to 10 VMs before scaling to 220.
3. How would you finish the remaining 220 VMs within two hours without degrading the Prism Central UI?
Read the worked answer
Reasoning: Start by measuring the sustainable request rate with a small batch, then set concurrency below the point where 429s appear. Use a queue with backoff, not a fixed thread pool that hammers the VIP. Run the bulk job in a staggered manner and monitor Prism Central CPU, memory, and API latency; if UI users report slowness, reduce the rate rather than increasing retries.
Before the run, reconcile the existing 300 names and remove or fix duplicates and partial VMs. Use idempotent naming and skip already-complete VMs. Poll tasks in batches rather than one poll per second per VM. If two hours is still tight, ask whether the deadline can move or whether the job can run outside peak hours; do not sacrifice correctness by disabling backoff or ignoring task states.