Halting a $180,000-a-Day Line to Stop a Ransomware Spread

Sever the link, deliberately, and document it as an incident-commander decision.

The technical case is not close. Plant 2's controller shares a flat VLAN with an office segment that has already produced an infected host. There is no segmentation control that can be implemented within the hour with confidence - a VLAN change on production OT without a tested design is its own outage. Leaving the link up accepts an uncontrolled spread path from IT into OT, where the consequences are not a late-delivery penalty but safety events, physical damage, and a multi-week outage across all three plants.

The commercial argument is bounded, the OT argument is not. $180,000 per day of revenue plus $50,000 per day of penalty is a quantifiable, recoverable loss with a contractual counterparty you can talk to. An OT incident is unquantifiable, potentially injures people, and - critically - you have no proven recovery: the only backups are 11 months unverified and sitting on a domain-joined NAS that the attacker can already reach. "We can restore" is not an available answer, so the downside is not capped.

Handle the pressure explicitly. The incident commander owns the security call; the business owns commercial acceptance. Put the tradeoff in writing, escalate to the executive owner, and give them a decision window measured in minutes with the default stated: "absent an explicit override, I isolate plant 2 at 10:15." An override is recorded with a name against it. Never let the call be made implicitly by silence.

Immediately after: image the controller state before touching it, protect the backup NAS (disable domain trust, disconnect, verify any offline copy), and notify the insurer inside the 24-hour window.

Order matters more than speed.

1. Stop the spread before restoring anything. Isolate affected segments, disable compromised accounts, and - if domain administrator compromise is suspected - treat the domain itself as untrusted and follow a documented domain recovery procedure rather than resetting one password and hoping.
2. Secure the recovery capability first. The NAS is domain-joined and reachable; it is a target, not an asset. Disconnect it, verify whether an offline or immutable copy exists, and confirm the integrity of the backup data before relying on it. Restoring onto an infected network reproduces the incident with extra steps.
3. Identify and remove the persistence vector. Restoring without knowing the initial access and the persistence mechanism produces re-infection, typically within days.
4. Rebuild, do not clean. Known-good images, unique local administrator credentials per host, EDR healthy and current on every rebuilt system.
5. Restore in a phased order: least-exposed plant first, non-production isolated segment first within it.

Gates that must all pass before a plant reconnects:

• No new encryption events across all monitored shares for a defined quiet period, with monitoring confirmed to be working rather than merely quiet.
• The persistence vector is named and removed, evidenced - not assumed.
• ERP data integrity verified against a known business checkpoint, not just "the service started".
• OT controller firmware and configuration verified against a known-good baseline and tested in isolation.
• The flat IT/OT VLAN is segmented, or the OT segment is otherwise isolated, before reconnection. Reconnecting to the same flat network re-creates the condition that forced the halt.

Add a business sign-off and a rollback plan for each phase. Recovery is a series of reversible steps, not a switch.

Judge the decision against the information available on Tuesday morning, not against the outcome. If no OT infection occurred, the wrong conclusion is "the halt was unnecessary". The right question is whether the information then available justified it and whether the cost estimate was accurate.

Questions for the review:

• Why were OT and IT on a flat VLAN, and what control gap allowed an office host to sit on the same segment as a line controller? Who owned that decision?
• Why did a six-figure-per-day production halt have no pre-agreed decision rule? The absence of a playbook turned an operational call into an ad hoc executive negotiation and cost an hour.
• What did the halt actually cost, versus the estimate used in the room? Track this, because next year's decision will be calibrated on it.
• Was the insurer notified within 24 hours, and was the customer told - and did the penalty clause actually trigger, or was it waived in negotiation?
• Was the 11-month-old restore test known to the incident commander at the time? If not, the review has found a communication failure as well as a backup failure.

Output should be a pre-authorised rule, not a paragraph: "the incident commander may isolate any OT segment without prior business approval, and will notify within 30 minutes." That converts a one-hour negotiation into a thirty-second action, and is the only finding from this incident that reliably improves the next one.