Ransom Note on the Shop Floor: Containment at $180,000 an Hour
You are the on-call security manager at a manufacturer with two plants running a 24/7 line. At 21:40 the EDR console shows the same beaconing pattern on three servers: the MES application server that orchestrates the assembly line, the backup catalog server, and a file server holding 2 TB of engineering drawings. A help-desk ticket at 21:55 reports a ransom note printed on a shop-floor printer. The MSSP that triages alerts has a 15-minute SLA but no authority to isolate anything; its playbook says to open a ticket and wait.
The night plant manager says stopping the line costs about $180,000 an hour and a customer shipment leaves at 06:00. Backups run nightly to a NAS in the same data center and weekly to tape that goes offsite; the last restore test was 14 months ago. Nobody has verified whether the NAS is reachable from the MES server.
1. What do you do in the first 30 minutes, and who decides whether the MES server is isolated?
Read the worked answer
21:40–21:50 - protect the recovery path first. Ask one question: can the MES server reach the backup NAS? If yes, assume the backups are being encrypted right now. Sever the NAS from the server network immediately; that is a low-regret action with no production impact. Then block the beacon's command-and-control destination at the egress firewall, and disable any account observed authenticating from the three hosts.
Contain by network, not by power. Isolating a host at the switch or EDR preserves memory and logs. Powering it off destroys evidence you will need and does not help recovery.
Who decides on the MES. The plant manager owns the production consequence; you own the security recommendation. Give a clear recommendation, not a menu: the MES is beaconing, so its output and quality records are no longer trustworthy, and continued operation may be pushing untrusted product. State the cost of both options - $180,000 an hour versus shipping suspect product and rebuilding a corrupted line. Escalate to the incident commander and an executive now; do not let a 21:40 call be made by the on-call analyst alone.
Do not wait for the MSSP. Its 15-minute SLA buys you a ticket, not containment. Open a bridge, page the team, and treat the MSSP as an evidence source.
21:55 ransom note confirms encryption is in progress somewhere - hunt for the encryption parent process across all hosts before doing anything else.
2. At 02:00 the line is stopped. Do you restore from backup or rebuild, and what do you validate before reconnecting anything?
Read the worked answer
Assume the worst about both backup sets until proven otherwise: the NAS was in the same data center and possibly reachable from a compromised host; the tape is offsite but has not been test-restored in 14 months, and an untested backup is a hypothesis.
Order of operations.
- Inventory the tape vault and confirm the last good backup date pre-dates the earliest evidence of compromise, not the earliest alert.
- Restore to an isolated network segment - not the production VLAN - starting with the backup catalog and one non-critical server. Verify the catalog itself is not encrypted or tampered with.
- Rebuild rather than restore for the three confirmed-compromised hosts. Restoring a server from a backup that contains the intruder's persistence just restarts the clock. Rebuild from known-good images or media.
- Reconnect only after the credentials on those hosts are rotated, the domain has no unexplained accounts or scheduled tasks, and EDR confirms clean for 24 hours.
Validation before reconnection. Confirm backups are immutable or offline going forward; scan the restored data for the ransomware extension; test that the MES can talk only to its required endpoints, not the file server or the NAS; and have the plant run a short trial batch that quality signs off on.
On the 06:00 shipment: tell the customer the truth early. A missed shipment is a commercial problem; a shipment built on a compromised MES is a safety and liability problem.
3. What do you change about the MSSP arrangement and internal authority after this event?
Read the worked answer
The failure was not detection speed - the EDR caught the beacon. It was that the only party watching at 21:40 had no permission to act, and no one had pre-decided that stopping the line is preferable to running a compromised controller.
Delegate containment authority with hard boundaries. Write a playbook that authorizes the MSSP to isolate a host matching specific high-confidence criteria - confirmed C2 beaconing, ransomware extension, known-bad hash - without prior approval. Scope it narrowly enough that a false positive costs one reboot, and require notification within 15 minutes of acting. Add a mutually agreed definition of "confirmed" so the MSSP is not guessing.
Pre-decide the business call. Get an executive-signed statement now that an MES server showing beaconing is isolated regardless of production impact, with the plant manager notified in parallel. At 21:40 nobody should be calculating cost per hour for the first time.
Technical changes: segment OT from IT so the MES cannot reach the file server or the NAS; make backups immutable and offline-copy; test a restore quarterly and record the result; put the backup catalog on separate credentials and a separate network path.
Run a tabletop with the plant manager, the MSSP, and communications in the room, using this exact timeline. The next incident will not be this one, but the decision structure transfers.