Playbook Fails at Containment: Rate Limits, Reruns, and a Dead Handoff
A Cortex XSOAR playbook named 'Enrich and Contain' processes about 300 incidents per day. It has eight parallel enrichment branches, then calls an EDR integration to isolate the endpoint, creates a ticket, and posts to a chat channel. Over the last month, 11% of runs fail at the containment stage. Logs show HTTP 429 responses from the EDR API during peak hours; the playbook retries three times with a fixed two-second delay and then stops. The EDR vendor documents a limit of 60 requests per minute per tenant. Some endpoints were isolated twice because analysts reran the playbook manually. Sixty incidents sit in 'Awaiting user input' because the reviewer task is assigned to a group with no weekend members. The ticket connector has no deduplication, so reruns create duplicate tickets. Leadership wants zero failed playbooks next quarter.
1. Diagnose the failure modes and propose a redesigned playbook structure.
Read the worked answer
There are three distinct failure classes here, not one reliability problem.
Rate limiting. Eight parallel branches across 300 incidents per day produce bursts well above 60 requests per minute. A fixed two-second retry is the wrong pattern: it synchronizes retries and amplifies the burst. Use exponential backoff with jitter, honor Retry-After, cap concurrency per integration instance, and route containment calls through a rate-limited sub-playbook. Classify 429 and 5xx as retryable, and authentication or validation errors as terminal so they fail fast to a human.
Non-idempotent containment. Check the endpoint's current isolation state before calling isolate, and record an action key built from incident ID and endpoint ID so a rerun is a no-op. Apply the same idea to the ticket connector: dedupe on incident ID, not on title.
Human handoff. A group with no weekend members is a design error. Replace it with a role-based assignment, an SLA timer, and escalation to on-call, or make the task optional with a default decision and a 30-minute timeout.
Challenge the zero-failures target. Some failures are correct, such as refusing to isolate a domain controller. Aim for zero silent failures and bounded time to containment instead.
2. How would you test and roll out the redesign without risking production containment?
Read the worked answer
Build the harness before touching production. In a dev tenant, replace the EDR integration with a mock that returns 429 for the first two calls, then success, then a terminal 400. Assert three behaviors: the playbook backs off rather than retrying immediately, it does not issue a duplicate isolation call, and terminal errors route to a human task instead of looping. Replay roughly 200 historical incidents with recorded API responses to catch context-shape regressions.
Rollout in stages. Feature-flag the new version, enable it for one low-risk incident type, and compare failure rate, containment latency, and duplicate-isolation attempts against the old version. Keep the previous version one click away, because playbook rollback during an active incident is not the time to rebuild.
Monitor the 429 rate, retry count, time spent in 'Awaiting user input', and duplicate tickets created. Define a rollback trigger in advance - for example, containment latency above 10 minutes or any duplicate isolation - and agree it with the SOC lead before enabling, so the decision is not made under pressure at 03:00.