Agentic AI Guardrails for Destructive Recovery Runbooks
An enterprise deploys an agentic AI assistant to automate recovery runbooks across three business units. The assistant can query backups, initiate restores, and modify SLA policies. Of 220 runbooks, 40 are marked 'destructive' (delete snapshots, failover clusters). RBAC has five roles: viewer, operator, approver, admin, auditor. The AI agent runs under a service account with operator rights. In a pilot, the agent proposed restoring a 12 TB database during business hours, which would have saturated the 8 Gbps replication link. Auditors require every agent action to be attributable and reversible where possible. The platform logs agent prompts and actions but not the model version or confidence score. Business units want faster recovery; security wants no autonomous destructive actions.
1. Design guardrails for the agent's 40 destructive runbooks. Be specific about roles, approvals, and execution conditions.
Read the worked answer
Destructive runbooks must never execute autonomously under the current operator service account. Create a separate 'propose-only' service identity with read and draft rights, and require the approver role to authorize execution. For the 40 destructive runbooks, enforce a two-person rule: the agent proposes, a human approver with the approver role reviews the plan, and an admin or operator executes or releases a just-in-time elevation. Add change-window and blast-radius checks before execution: the agent must show affected VMs, replication impact, and rollback plan. Block execution if the proposed action would consume more than a defined percentage of the 8 Gbps link during business hours. The platform should default-deny any runbook not explicitly tagged as non-destructive. Auditors should see the approver identity, timestamp, model version, and the exact parameters.
2. How should the platform ensure attribution and reversibility for every agent action?
Read the worked answer
Attribution requires logging more than the prompt and action. Record the model version, confidence score, retrieval context, proposed runbook, parameters, approver, execution identity, and outcome in an append-only audit log. Correlate the agent session ID with the change ticket and the backup job ID. Reversibility requires designing runbooks as idempotent and reversible steps: take a pre-action snapshot or configuration backup, use dry-run or simulation mode first, define rollback commands, and verify state after each step. For actions that cannot be reversed, such as deleting the only snapshot copy, require explicit acknowledgement and a second approver. The agent should not be allowed to modify its own audit logs or RBAC. Test rollback quarterly and include failed rollbacks in the incident process.
3. What metrics and controls would you use to detect agent drift or unsafe proposals?
Read the worked answer
Monitor leading indicators, not just restore success. Track the rate of agent proposals that are rejected or modified by approvers, the percentage of proposals that violate change-window or bandwidth constraints, the number of destructive runbooks proposed without rollback plans, and the difference between agent-selected backup points and human-selected points. Compare model versions against a golden set of 30 known scenarios and require re-approval after any model update. Add anomaly detection for unusual runbook selection, such as a sudden spike in failover proposals, and alert on confidence-score distributions shifting downward. Review a sample of agent prompts and rationales weekly, and feed rejected proposals back into prompt and runbook tuning. Business-unit recovery-time metrics should be paired with a safety metric so speed gains do not silently increase risk.