A Monthly Snapshot Cleanup Runbook That Cannot Finish in Its Window
A runbook runs on the first day of each month at 01:00 to snapshot 240 VMs and delete snapshots older than 30 days. It is built as a single serial sequence: for each VM, create a snapshot, wait 60 seconds, list snapshots, delete each old snapshot, then write a line to a log file on a file server. The maintenance window is 4 hours. Last month it ran for 9 hours, overlapped with business hours, and 14 VMs were skipped because the runbook hit an execution timeout. Three deletions failed with 'snapshot in use' but the runbook continued and reported success. The log file write also failed after the file server ran out of space, yet the runbook status stayed 'SUCCEEDED'. The team wants to use the same runbook to call a REST endpoint that tags each VM with the last snapshot date. The API returns HTTP 202 with a task ID.
1. Why did the runbook take 9 hours and miss 14 VMs, and how would you restructure it to fit a 4-hour window without overloading Prism Central?
Read the worked answer
The runbook is serial and pays a fixed 60-second wait for every one of 240 VMs. Even without failures, that is 240 minutes of waiting, plus snapshot, list, delete, and log tasks. The 9-hour runtime and timeout are therefore structural, not a transient slowdown.
Restructure into bounded parallel batches, for example 20-30 VMs at a time, with a concurrency cap based on Prism Central and cluster load. Replace the fixed wait with polling of the snapshot creation task until it reaches a terminal state, then list and delete. Keep per-VM work independent so one slow VM does not block the batch. If the runbook engine does not support parallel foreach, split the runbook by cluster or category and stagger start times, or call a script that implements bounded concurrency. Add a pre-check that skips VMs already snapshotted in the current cycle. Measure a pilot batch's duration, then set the batch size to finish with headroom inside four hours. Do not simply raise the timeout; that extends the business-hours overlap.
2. Three deletions failed with 'snapshot in use' but the runbook reported success. How should error handling and reporting change?
Read the worked answer
The runbook treated a failed delete as non-fatal and continued, then reported the overall run as successful because the sequence completed. That hides partial failure. Each VM should be a unit of work with an explicit result: snapshot created, old snapshots found, each deletion succeeded or failed, and final status.
Classify 'snapshot in use' as retryable with a short backoff, because a clone or backup may still hold the snapshot; after the retry budget, mark that VM as failed and preserve the error. Apply a policy: continue processing other VMs, but fail the runbook at the end if any critical deletion failed, and emit a summary with failed VM UUIDs. Make snapshot creation idempotent by using a deterministic name and timestamp so a retry does not create a second snapshot. Do not write to a single file-server path without checking space; use structured runbook output or a log target with retention, and fail loudly if logging is part of the compliance requirement. Reconcile the 14 skipped VMs explicitly rather than assuming the next run will catch them.
3. The runbook must call a REST endpoint that tags each VM with the last snapshot date and returns HTTP 202 with a task ID. What integration pattern do you use?
Read the worked answer
HTTP 202 means the request was accepted asynchronously; it is not proof that the tag was applied. The runbook must capture the task ID from the response, poll the task endpoint (for example /api/nutanix/v3/tasks/{uuid}) with backoff until it reaches SUCCEEDED or FAILED, and only then record the VM as tagged. Do not tag before the snapshot task has succeeded.
Use a dedicated least-privilege service account, keep credentials out of the runbook body, and handle 401 and 403 as configuration failures that stop the batch. Add rate-limit handling for 429 with Retry-After and bounded retries; a 202 followed by a failed task is a real failure, not a success. Store the task ID and final state per VM so a rerun can skip already-tagged VMs and reconcile in-flight tasks. If the runbook engine cannot poll, have it call a script that does the polling and returns a single status, but keep the same idempotency and audit requirements.