A Remediation Playbook That Keeps Re-Firing
An administrator creates a Playbook in Prism Central to remediate a 'Memory usage high' alert on 18 VMs in a production project. The trigger is the built-in alert; the actions are: add 4 GB memory, reboot the VM, and post a message to a team chat webhook. During a load test, the alert remains active for 25 minutes on one VM. The playbook runs five times, rebooting the VM each time; the VM is now in a reboot loop and the chat channel has duplicate messages. The webhook action also fails intermittently with HTTP 429. The admin wants to add a 10-minute delay at the start of the playbook and increase the webhook retry count. The application owner is concerned about downtime and asks for an approval step for production. The Prism Central project has 12 users, two of whom are application operators who should be able to approve but not edit the playbook.
1. Diagnose why the playbook ran five times and rebooted the VM repeatedly. Why is a 10-minute delay at the start and a higher webhook retry count not a sufficient fix?
Read the worked answer
The alert stayed active for 25 minutes, so the alert trigger evaluated repeatedly and started new playbook runs. The playbook had no incident-level deduplication, no check that a run was already in progress, and no condition that the VM's memory was still below target. Rebooting does not clear a memory-pressure condition; the next evaluation saw the same alert and remediated again. The webhook 429 likely caused action retries, and if the retry re-entered the action or the playbook, it amplified duplicates.
A 10-minute delay only widens the window; the alert can still be active after the delay, so the same loop returns. Increasing webhook retries makes duplicate notifications more likely unless the webhook is idempotent. Fix the trigger and state model: run once per alert incident, lock or mark the VM as 'remediation in progress', check current memory and whether a reboot already happened, and wait for the alert to clear before closing. For production, require approval before the reboot. Treat notification failure separately from remediation failure.
2. Redesign the playbook flow so it is safe for production and still handles the chat webhook's HTTP 429 responses.
Read the worked answer
Proposed flow:
- Alert trigger with incident deduplication and a condition that the alert is still active and the VM is in the production category.
- Approval action for production, with a timeout and explicit rejection path.
- Re-check current memory and remediation state after approval, because conditions may have changed.
- If memory is still below target, perform the approved action (add memory and reboot) and wait for the Prism task to complete.
- Wait for the alert to resolve or timeout, and record the outcome.
- Send the chat notification as a non-blocking action.
For 429, use exponential backoff with jitter, honor Retry-After, cap attempts, and include a deterministic idempotency key such as alert ID plus VM UUID so duplicate sends collapse. Do not let a failed notification retry the reboot. If the playbook engine cannot express 'run once per incident', use a manual trigger or an external incident ID, or suppress the alert after the first run. Document that adding memory may itself require a reboot, so sequence it deliberately.
3. The two application operators must approve production remediations but must not edit the playbook. How should access be structured, and what would you verify?
Read the worked answer
Do not give the operators the playbook-editor role. Assign them a project-scoped role that includes the approval permission and read/view access to the playbook and its runs, but excludes create, update, and delete on playbooks and blueprints. Keep the playbook owner/editor group separate and small.
Verify that the approval action resolves to the project role, not to a global Prism Central admin role; otherwise the operators may silently gain broad rights. Check that approval notifications reach both operators and that a single rejection stops the run. Test with a non-production playbook: operator can approve or reject; operator cannot edit or disable the trigger; auditor can read run history but not approve. Also confirm the service account that executes actions is not the operators' account, and that the approval record captures the operator identity, timestamp, and decision in the audit trail.