The Playbook That Isolated a Production Database
A SOAR playbook isolates any endpoint whose EDR verdict is 'Credential Dumping', with no approval gate. At 14:05 UTC it isolated a production Oracle database server; the order management application failed over incorrectly and was unavailable for 22 minutes. The triggering event was Sysmon Event ID 10 (ProcessAccess) with GrantedAccess 0x1010 against lsass.exe by a backup agent's service binary. The CMDB record for the server still lists asset type 'workstation' because the record was never updated after a 2023 migration, and the playbook's exclusion list keys on that field. The tenant API allows 600 calls per minute, and the playbook's enrichment loop issues roughly 900 whenever a full asset lookup misses the cache. Forty active workflows share the tenant. Post-incident review is tomorrow and the platform owner wants a design, not a patch.
1. What do you fix first: the approval gate, the CMDB data, or the detection logic?
Read the worked answer
The trust boundary, not the gate.
The playbook did what it was told. The CMDB said 'workstation', so the isolation path ran. Adding a gate first would slow all forty workflows and would still isolate this server the moment a human clicked through, because the approver would read the same stale record.
Fix in this order:
- Remove the dependency on a field that can be silently stale. Criticality must come from a live source or resolve to 'unknown', and 'unknown' must never auto-contain. A record untouched since before a migration is a red flag, not a fact.
- Change the trigger. ProcessAccess to lsass from a signed backup agent binary with a documented GrantedAccess mask is a known-good pattern, not credential dumping. Exclude by signer and image, and alert rather than isolate.
- Only then add the gate, scoped to isolation actions and not to enrichment.
CMDB cleanup is a separate funded project. Do not make the playbook wait for it; make it fail closed in the meantime.
2. Design the approval gate so it prevents this failure without paralysing the SOC.
Read the worked answer
Risk-tiered, fail-closed on the unknown, and time-bounded.
- Tier 0 or 1, or unknown criticality: isolation requires approval from a named on-call owner with a 10-minute response SLA and automatic escalation to the SOC lead, so the request cannot expire silently. Enrichment still runs immediately, so the approver sees process tree, user, hash and asset owner.
- Tier 2 or 3 endpoints whose record changed within 90 days: auto-isolate, notify the owner, and auto-release after a maximum window unless renewed.
- Break-glass only for confirmed active ransomware on a host, with mandatory post-hoc review.
- Every decision, including auto-approvals, writes an audit record containing the data it relied on, so the next review can tell whether the gate or the data failed.
Measure it afterwards: approval latency and false-isolation rate. A gate that adds hours to containment is simply a different failure mode from the one you just had, and the CISO will notice.
3. The enrichment loop can issue 900 calls a minute against a 600 limit. How do you make that safe?
Read the worked answer
Budget the tenant explicitly rather than hoping the retries cope.
- Assign each workflow a declared call budget at design time and enforce it in the platform, not inside a script. Forty workflows sharing 600 calls per minute is roughly 15 each; a loop issuing 900 has no budget at all.
- Cut the calls: batch asset lookups, use bulk endpoints, cache CMDB and hash reputation with TTLs matched to data volatility, and skip enrichment for fields the decision does not use.
- Make failure loud. A throttled lookup currently returns nothing, and the playbook reads silence as 'not critical'. Instead, mark the case enrichment-incomplete and block any auto-containment decision on partial data.
- Test the failure path deliberately: inject a 429 and confirm the workflow halts rather than proceeding.
Rollback needs the same rigour. Verify the unisolate action is idempotent, that isolation failures are detected rather than assumed successful, and that a partially isolated host is treated as an open incident. Add the rate-limit scenario to the playbook's test suite.