Water Utility OT: Re-Enabling IPS and Segmenting a Flat Plant Network Without Downtime

The plant must keep running, so containment happens at the switch and the policy layer, not by shutting the OT link or rebooting devices.

1. Isolate the vendor laptop at the port, not by chasing the host. Use FortiSwitch port-level controls to disable the specific access port or move it to a quarantine VLAN, then find the laptop's MAC and confirm it is no longer reachable from the historian or any PLC. If the worm has spread, port-level isolation is the only mechanism that works on unmanaged legacy hosts.
2. Do not isolate the IT-OT link. Cutting it would remove remote monitoring and any vendor support path while doing nothing about a worm already inside the flat OT VLAN. It also creates a safety and compliance problem if telemetry to the regulator stops.
3. Reduce lateral movement with what you already own. Put an interim policy on the 60F blocking SMB, RDP, and other IT-side protocols from crossing into OT except from the two named jump hosts; disable unused switch ports; enable port security with MAC limits; and check for DHCP snooping and ARP inspection where the switches support it.
4. Hunt passively. Take packet captures at the OT uplink and review FortiSwitch and FortiGate logs for the worm's characteristic traffic, rather than running active scans against PLCs - active scanning of Modbus devices can cause controllers to fault.
5. Engage the vendor and the plant manager, document every action with timestamps, and open an incident record that the insurer will eventually see. A written, ordered timeline is half the evidence requirement.

Do not re-enable IPS on the IT-OT policy in the first 24 hours unless you can tune it first; a second false-positive-driven outage during an active incident destroys whatever credibility you have left with the OT engineer.

The engineer made a rational decision: a security control was breaking production, so the control was removed. To reverse it you must remove the reason, not argue about policy.

• Reproduce the false positive deliberately first. Mirror a PLC polling session at the OT uplink, run it against the profile in monitor mode, and identify exactly which signature fires on which byte pattern. Modbus traffic from legacy PLCs frequently trips generic protocol-anomaly signatures because the devices do not follow the RFCs strictly.
• Build a Modbus-aware profile rather than the default. Disable the specific offending signatures by ID, not the whole category, and keep the signatures that catch real worm behaviour - SMB exploit and reconnaissance families matter far more here than protocol-anomaly signatures on deterministic polling.
• Stage it: monitor mode on the IT-OT policy for one full monthly cycle including a maintenance window, then packet-mode or block mode for non-critical signatures only, then full profile. Keep a documented rollback that the OT engineer can execute alone, because if they cannot roll it back they will disable it again.
• Virtual patching for the Windows 7 HMIs. They will not be patched - they are out of vendor support. Compensate with IPS signatures and strict policy between the HMI segment and the PLC segment, and record that as the accepted control in the risk register.
• Give the OT engineer a named signature review each quarter so the false-positive risk has an owner and a review date rather than being discovered in production.

Insurer evidence here is the profile configuration, the monitor-mode logs, and the rollback procedure - not just a statement that IPS is enabled.

Sixty days rules out anything needing plant downtime or re-IP of controllers. Work with what can be changed live.

Segmentation:

• Split the flat OT VLAN into three: PLC/controller segment, HMI/historian segment, and OT management/vendor segment. PLC IP addresses stay unchanged and identical at the controller side; only the switch port VLAN assignment and the gateway policy change, which can be done port by port with a person at the plant verifying that each control loop still polls. Do two ports at a time, in the monthly window.
• Policy between segments is deny-by-default with explicit allows: HMI to PLC on Modbus only, historian to PLC on Modbus only, management to everything only from the jump host. No IT-initiated traffic into the PLC segment at all.
• Use FortiNAC for visibility and profiling on the OT side rather than enforcement, initially. It gives you an asset inventory and detects rogue devices like the vendor laptop, which is directly what the insurer is asking about. Enforcement can follow once profiling is proven.

Vendor access:

• No direct OT port access. Vendors come in through a jump host in the management segment with MFA, session recording, and a time-boxed account. That single change addresses the actual incident.
• Every vendor session is scheduled, approved, and supervised by the plant engineer, with the FortiGate policy for the vendor enabled only for the session duration.

Deliberately exclude the corporate endpoint agent from the Windows 7 HMIs - it will destabilise unsupported machines and buy little. Compensate with segmentation and virtual patching, and document that as an accepted risk with the plant manager's signature. The insurer wants a defensible argument, not a perfect one.