Ransomware at a Manufacturer with a Live Furnace
Crestline Manufacturing runs six plants and 2,100 endpoints on a single Active Directory forest. At 02:14 on a Tuesday the EDR console raises credential dumping on a plant engineering workstation, followed by SMB lateral movement; by 05:00 forty servers are encrypted, including ERP and the manufacturing execution system. The nightly backup writes to an on-site NAS that replicates to cloud storage with a four-hour lag; immutable snapshots are retained for seven days. One plant's engineering workstation is dual-homed between corporate and OT networks. A furnace at Plant 3 needs a controlled six-hour cooldown or its lining cracks, and the operator HMI is unavailable. The attackers left a ransom note threatening to publish 3 TB of data, and egress logs suggest exfiltration began four days earlier. The CEO wants restoration started immediately, the insurer requires its own forensic firm to lead, and the CFO asks whether paying is faster.
1. Sequence the first six hours. What happens first and why?
Read the worked answer
Priorities in order: safety, spread, evidence, recovery.
- Safety (0-60 minutes). The furnace outranks IT. Bridge Plant 3's process engineer to the local control system using manual or local control, or execute the documented manual cooldown procedure, and record the decision. No containment action may remove the operator's ability to control a physical process.
- Declare and assemble (0-60 minutes). Incident commander named, war room opened, insurer and forensic firm engaged so that privileged direction arrives before destructive choices are made. Counsel engaged to direct the investigation where privilege is available.
- Contain the spread (1-3 hours). Isolate the dual-homed workstation and its OT segment first - it is the bridge. Disable compromised accounts, reset privileged credentials, block known C2 infrastructure at egress, and segment remaining plant networks. Do not mass power-off: encrypted servers yield nothing, but live hosts hold volatile evidence.
- Protect the backups (1-2 hours). Verify immutability is real, rotate backup credentials away from the domain, and confirm the cloud copy was not reachable with compromised credentials. This is the single action that determines whether recovery is possible at all.
- Preserve evidence (2-6 hours). Memory capture on key servers before reboot, disk images in a prioritised queue, and a chain-of-custody log.
- Assess and communicate. Confirm the exfiltration scope, brief the CEO that restoration cannot start cleanly until the persistence mechanism is identified, and prepare holding statements.
2. How do you decide when and in what order to restore?
Read the worked answer
Restoration is an evidence question before it is a speed question. Restoring onto a network that still contains the attacker's persistence simply re-encrypts the data and burns the immutable snapshots.
Preconditions: the forensic team has identified initial access and persistence; the domain has been rebuilt or cleaned, including a double krbtgt reset; credentials have been rotated; and the backup catalogue has been verified as free of the malicious binaries or scheduled tasks that may have replicated into it.
Sequencing by business impact, not by convenience:
- Plant 3 control and safety systems, once the OT path is clean and independently validated.
- The manufacturing execution system, because ERP without MES does not ship product.
- ERP and shipping/logistics functions, then corporate productivity systems last.
Method: restore into an isolated recovery VLAN, boot and scan, confirm clean, then promote. Restore in waves so an unnoticed reinfection costs one wave, not the estate. Validate data integrity for financial records, not just service availability - ERP restored to a four-hour-old state may silently duplicate or drop transactions.
On payment: paying does not shorten recovery, funds criminal operations, may breach sanctions law, and frequently results in a second extortion demand. Counsel and the insurer decide the negotiating posture; operations should plan on the assumption of no decryption key, and test that assumption against the recovery time objective. Tell the CFO plainly that the fastest route to production is clean backups plus rebuild, not a key.
3. Where do evidence preservation and recovery conflict, and how do you manage the conflict defensibly?
Read the worked answer
Every hour spent imaging a server is an hour not spent restoring it, and every restart destroys volatile state. The conflict is real and should be managed as an explicit, documented trade-off rather than resolved informally by whoever is loudest.
Management approach:
- Define what evidence is actually needed. For regulatory, insurance and potential litigation purposes you need to show initial access vector, scope of compromise, data accessed, and timeline. You do not need a bit-image of every endpoint. Prioritise domain controllers, the entry-point host, the dual-homed OT workstation, and any server that handled exfiltrated data.
- Capture volatile data first, then restore. Memory capture takes minutes and is irreplaceable; disk imaging can often run in parallel from a cloned volume while the original is rebuilt.
- Use parallel workstreams. The forensic team works on copies; the recovery team rebuilds from known-good media. They do not queue behind each other.
- Document every deviation. If a server must be wiped before imaging, record who authorised it, when, and why. A documented, reasoned decision survives scrutiny; an undocumented shortcut looks like spoliation.
- Chain of custody from hour one, with named custodians and hash verification for every image.
- Counsel decides privilege questions; operations should not label documents as privileged on their own initiative.
Finally, preserve attacker communications and the ransom note intact - they are evidence and may matter for sanctions screening, not merely for negotiation.