Auto-isolation that took out a domain controller

Automate what is reversible and low blast radius:

• Case creation for critical and high alerts. Cases are additive and carry no availability risk.
• Slack and paging notification, including a follow-up page if a critical alert has no owner after 15 minutes.
• Host isolation only for host.os.type: windows, only for hosts on a workstation value list, and only for malware-prevention alerts, not for behaviour rules.

Keep manual:

• Anything on a domain controller or production server.
• Anything on a server during the freeze window unless severity is critical and the on-call explicitly approves.

Mechanism: rule response actions are attached per rule and there is no built-in approval step, so the enforcement has to be structural. Clone the malware rule into two variants - one scoped to the workstation value list with the isolation action attached, one scoped to servers and DCs that only opens a case and pages. Keep the isolation action off the base rule so a future value-list edit cannot silently widen it.

Add two safety checks: a rule that alerts when a response action fails or times out, and a periodic review of which alerts were auto-isolated and later closed as false positive. If that count is not near zero, tighten scope before expanding automation.

The endpoint side is why it broke production; the rule side is why it alerted. Fix both.

1. Endpoint prevention: add the backup agent's binary and its install directory to the malware policy's trusted applications and exclusions so the file is not scanned or blocked. This stops the prevention alert at the source and keeps backups running. Accept and document the risk: anything living in that path is now unscanned, so the path must be narrow, owned, and re-reviewed quarterly.
2. Rule side: an exception for that exact binary path on the server and DC set, plus the workstation/server rule split from the previous step.

Order matters. A rule exception alone would have prevented the alert but not the block - the endpoint would still have blocked the backup agent, and you would have lost the evidence while keeping the outage. Conversely, a trusted application alone would work here but leaves no safety net if a different legitimate binary trips the same rule, which is what the server-side case-plus-page variant is for.

Both changes need change records with the path and hash, so the next person can see why that directory is excluded.

Response actions are asynchronous and can fail invisibly. For each automated action:

• Verify the outcome in the endpoint action response data stream and in the alert's own response-action fields. A timed-out action leaves the alert open and the host online.
• Write a rule that alerts on failed or expired response actions. A silently failed isolation is worse than no automation, because the analyst assumes containment and stops looking.
• Test against an offline agent and an agent on an older policy revision before rollout.

Case hygiene:

• Use a case template with required fields for scope, containment decision and close reason (true positive, false positive, duplicate, benign). Without a close reason taxonomy, reporting degenerates into counting closed cases.
• Assign from the alert owner field, tag by rule name, and move status to In progress when the Slack thread starts, so MTTD/MTTR is computed from real timestamps rather than from when someone remembered to edit the case.
• Keep the isolation timestamp in the case; the gap between alert time and isolation time is the metric leadership actually wants, and it is not derivable after the fact if nobody recorded it.

Finally, publish a weekly count of auto-isolations later closed as false positive alongside the alert volume. Automation that reduces alerts but increases bad isolations will be switched off by the infrastructure team within a month.