Scaling a Phishing Playbook Through a 300-Incident Burst
Harbor Financial's phishing playbook runs automatically for every incident of type 'Phishing.' It extracts URLs, detonates attachments in a sandbox, queries the email gateway, and disables compromised accounts in Active Directory. The email gateway integration allows 100 requests per minute; the sandbox takes up to 3 minutes per file. During a campaign, 300 phishing incidents arrive in one hour. Tasks begin timing out, the same account is disabled multiple times, and 60 incidents sit in 'Pending' because a manual approval task waits for an analyst. The integration instance uses one API key that expired last week, and no one noticed because the playbook's error path silently closed incidents as 'resolved.' The SOC operates 24x5, with 6 analysts per shift.
1. Redesign the playbook for the 300-incident burst. Identify which tasks can run in parallel, which need throttling, and how to make account disable idempotent.
Read the worked answer
Reduce work before adding concurrency. Add a classifier/mapper and a dedupe step that groups incidents by sender, URL, attachment hash, and campaign window. If 300 reports describe 40 unique messages, create or link to one parent investigation and attach the reports as affected users rather than running full detonation 300 times.
Parallelize independent lookups: URL reputation, sender authentication results, and attachment hash lookup can run together. Sandbox detonation is the long pole; run it only for unique hashes and cap concurrent sandbox submissions to the integration's limit. Throttle email gateway calls below 100 requests per minute, preferably with a queue or a task-level concurrency limit, and batch searches where the integration supports it. Do not let every incident independently poll the gateway.
Account disable must be idempotent: first query the account state, skip if already disabled, and use a stable idempotency key such as incident ID plus account SID. Record the result on the incident. If two incidents target the same account, the second should see 'already disabled' and stop. Use a short retry with backoff for transient errors, and route permanent failures to a manual task.
Keep the manual approval off the critical path: auto-approve low-risk containment where policy allows, and use a timer with a defined default for the rest. Never silently close on error.
2. How should the team handle the expired API key and the silent error path? What controls prevent recurrence?
Read the worked answer
Rotate the API key immediately in the credential store, not in the playbook text, and verify the integration instance uses the new credential. Confirm the key has least privilege for the gateway actions the playbook needs. Run a connectivity test command from the integration instance before re-enabling the playbook.
The silent error path is the more serious defect. Replace 'continue on error' on critical tasks with explicit failure handling: create a manual review task, set a visible incident field such as containment_status = failed, and notify the queue owner. Do not allow a task failure to flow into a 'close as resolved' branch.
Prevent recurrence with:
- A scheduled playbook that tests each critical integration with a harmless read-only call and opens a maintenance incident on failure.
- Monitoring for authentication failures and a dashboard for integration health.
- Credential expiry tracking and an owner for each integration instance.
- Playbook unit tests that include an expired-key response and assert that the incident does not close.
- Audit of error paths in existing playbooks, prioritizing any path that can close an incident without analyst action.
3. What would you change about the manual approval step?
Read the worked answer
Base approval on risk, not on incident count. Auto-approve containment when the message is clearly malicious, the account is not privileged, the action is reversible, and the sandbox verdict is conclusive. Require manual approval for privileged accounts, executive accounts, ambiguous verdicts, or actions that affect shared mailboxes or OT-adjacent identities.
Replace the unbounded wait with a timed task: assign the approval to the on-shift queue, set an SLA, and define what happens at timeout. For low-risk cases the default can be 'contain and review'; for high-risk cases the default should be 'do not contain, escalate.' Record the decision and the approver in the incident.
Measure approval latency separately from investigation time. If 60 incidents are waiting, the queue design is the problem: route approvals to a named queue with a shift owner, show the count on the SOC dashboard, and escalate when the oldest approval exceeds the SLA. After the campaign, review a sample of auto-approved cases to confirm the risk rules were correct, and tighten them if any containment was unnecessary.