Automating Faster Into the Plant Floor
Northgate Utilities runs a 12-person SOC across IT and OT. Its SOAR platform has a playbook called High = Isolate that quarantines any host with an EDR detection of severity High, with no approval and a median time to action of 40 seconds. During the monthly patch window, an EDR agent update changed how a third-party monitoring DLL injects into processes, and 34 hosts fired Suspicious process injection. Two were domain controllers; one was an HMI jump server on the plant floor. All 34 were quarantined within three minutes, the domain controllers lost network access for 22 minutes, and the plant's SCADA operator lost the HMI link for 40 minutes. The incident was ultimately benign. Separately, the team carries 1,100 untriaged alerts, and the threat-intel platform adds about 300 IOCs per day, of which roughly 40% are later judged low-confidence or irrelevant to the company's sector. The CISO's response to the outage is to automate more so humans do not slow us down.
1. Redesign the auto-isolation playbook. State when isolation should be automatic, when it should require approval, and what isolation should technically mean for IT versus OT assets.
Read the worked answer
The outage did not happen because a human was in the loop; it happened because the playbook had one input and no blast-radius model. More automation with the same design reproduces the outage faster.
Automatic, no approval when the detection is corroborated by a second independent signal (process injection plus a C2 network event, or a credential anomaly plus lateral SMB traffic), AND the asset is non-critical in the CMDB, AND it is not a domain controller, hypervisor, or OT asset, AND the action is reversible within minutes.
Requires approval with a five-minute escalation timer for crown jewels, domain controllers, and anything in the OT segment. The timer matters: approval queues that wait hours are how you get pressure to remove the gate entirely.
Never automatic on a single uncorroborated detection, and never during a declared change window for detections whose signature matches the change ticket. That is targeted contextual suppression, not a blanket patch-window mute.
What isolation means technically. Stop using full agent quarantine as the only tool. For IT, prefer switch-level ACLs or a restricted VLAN that blocks east-west and egress while preserving EDR management, patch, and backup traffic. For OT, never sever the HMI link: use a monitoring-only posture, alert the plant operator, and let a human with plant authority decide, because a 40-minute HMI loss can be a safety event, not an availability event.
Add a dry-run mode for every new playbook, a global kill switch, and a mandatory post-action report of what was cut off.
2. How would you work down the 1,100-alert backlog without bulk-closing risk?
Read the worked answer
First, stop the inflow so the backlog is a fixed quantity: identify the top five rules by volume and fix or re-tier them. If 90% of the backlog came from twenty rules, triaging the backlog without fixing the source is a recurring cost with no end.
Then triage as a sorted, grouped queue rather than 1,100 individual items:
- Group by entity and by campaign. Many of these are five alerts about one host or one rule firing repeatedly. Real work items are probably in the low hundreds.
- Rank by asset criticality × detection confidence × identity privilege × data sensitivity. Domain controllers, finance systems, and privileged accounts come out first regardless of detection severity label.
- Bulk-close only with a measurable justification. A rule family with a documented false-positive rate above an agreed threshold, an identified benign cause, and a compensating detection already in place may be closed in bulk. Sample-audit 5–10% of any bulk closure and record the sample.
- Never bulk-close detections on crown-jewel assets, detections with no assigned owner, or anything involving credential access, or the queue will be clean and wrong.
- Track backlog age, not just count. Median age of open alerts is the metric that shows whether the process is working.
Finally, feed the outcome back into detection engineering. A backlog is usually a detection-quality problem wearing a staffing costume, and the CISO should see both numbers side by side.
3. How do you handle 300 IOCs per day when roughly 40% are later judged low-confidence or irrelevant to the sector?
Read the worked answer
At 300 per day the team cannot adjudicate individually, so the pipeline must score automatically and only escalate what is actionable against your own telemetry.
- Score on four axes: source reliability (historical hit rate per feed, measured not assumed), freshness, relevance to sector and geography, and observability - whether the IOC type can even be seen by your sensors. A file hash is useless if you have no hash telemetry on the relevant hosts; an IP is useless if you do not log the traffic that would reach it.
- Auto-enrich, do not auto-block. Enrichment is cheap and reversible; blocking on a 40%-low-confidence feed will eventually block something the plant needs, and OT outages are not measured in minutes of annoyance.
- Give every block a TTL and an owner. Most intel-derived blocks should expire in 30 days unless renewed; stale blocks accumulate and become their own outage risk.
- Measure feed performance quarterly. Hit rate, false positives generated, and analyst time consumed per feed. Drop or downgrade feeds that do not earn their slot; 40% low-confidence suggests at least one feed is being ingested for completeness rather than value.
- Use intel primarily to generate hunts, not blocks. The highest-value output of a low-confidence feed is a hypothesis - a new C2 provider, a new loader - tested against 30 days of retained telemetry, rather than a rule that fires on incidental traffic.
Report intel performance to the CISO with the same rigour as backlog age, because the 300-per-day figure looks like coverage and often behaves like noise.