Ransomware at a Food Plant: Restore the Historian or the Line?
A food and beverage plant has two production lines. Ransomware encrypts corporate IT and the MES. The historian is domain-joined, and its backups sit on a domain-joined NAS whose catalog is also encrypted. Line 1 continues in manual with local PLC logic. Line 2 is down because the MES is required for recipe download. The OT network was not directly encrypted, but the domain controller used by the historian and engineering workstations is unavailable. The last known good historian backup is 31 hours old. The DR plan claims a 36-hour RTO and 4-hour RPO for critical manufacturing systems. The plant manager wants to restore a 5-month-old domain controller image to bring the historian back quickly. The cyber insurer requires a forensics firm before restoration. The next shift starts in 10 hours; product shelf life is 48 hours. A manual recipe download to line 2 is possible with a vendor engineer in 6 hours, but carries quality risk.
1. What recovery sequence do you direct, and why not restore the five-month-old domain controller image?
Read the worked answer
Sequence: stabilize line 1, recover line 2 manually, rebuild identity services cleanly, then restore the historian.
Why not the old DC image: a five-month-old domain controller reintroduces stale accounts, missing patches, and possibly the original compromise. It can also create replication conflicts and trust failures. Restoring it to 'get the historian back quickly' trades a short-term dependency fix for a longer, less trustworthy recovery.
Actions:
- Confirm line 1 manual controls and quality checks; do not destabilize a running line for an IT recovery.
- Authorize the vendor engineer to perform the manual recipe download to line 2, with quality hold points and a documented risk acceptance from the plant manager.
- Build a clean domain controller from installation media, not from the old image; restore only required accounts and trusts.
- Restore the historian from the 31-hour backup after validating the backup integrity and scanning it.
- Preserve forensic evidence in parallel; do not delay the manual line-2 recovery for imaging.
The 36-hour RTO is not credible until dependencies such as DNS, domain services, and the NAS are mapped.
2. What does this event reveal about the RTO/RPO claim and backup architecture, and what would you change?
Read the worked answer
The DR plan claims a 36-hour RTO and 4-hour RPO, but the historian backup is 31 hours old and the backup catalog is encrypted. The real RPO is at least 31 hours, and the RTO is undefined because recovery depends on a domain controller, DNS, and a domain-joined NAS.
Architecture changes:
- Move OT backups off domain-joined storage. Use immutable, offline or off-network repositories with separate credentials.
- Separate OT backup infrastructure from the IT domain so ransomware on IT cannot encrypt the catalog.
- Test restore of the historian, MES recipe data, and PLC programs, not just backup job success.
- Document dependencies: historian depends on DNS and domain authentication; line 2 depends on MES and recipe data.
- Set RTO/RPO by production line and product shelf life, not a single generic 'critical manufacturing' number.
A 4-hour RPO is a claim that must be proven by a restore test. If the business will not fund immutable OT backups, the risk owner must accept the gap in writing.
3. How do you reconcile the insurer's forensics requirement with the 10-hour shift start and 48-hour shelf life?
Read the worked answer
The insurer requirement does not override safety or product shelf life, but it should not be dismissed either. Reconcile by scoping forensics and running work in parallel.
- Scope forensics to IT and the domain-joined systems while preserving OT logs. The forensics firm can image the NAS, domain controller, and MES servers without blocking the manual recipe download.
- Preserve before rebuild: capture historian and engineering workstation logs, firewall logs, and PLC program checksums before restoration. Use write blockers where possible.
- Document the deviation: if the manual line-2 recovery starts before the forensics firm arrives, record why - shelf life, shift start, quality risk - and what evidence was preserved. Notify the insurer in writing.
- Do not restore the old DC to satisfy a speed requirement; that increases the loss and may jeopardize coverage.
Incident command should assign separate leads for production recovery, forensics, and insurer communication so one does not block the other.