Broadcast storm at a sewage pumping network - root cause and the control gaps behind it

Immediate cause: a Layer 2 broadcast storm caused by a physical loop. The unmanaged switch created a second path into the ring. With no spanning-tree or ring-redundancy protocol active on that unmanaged device, broadcasts and unknown unicast frames circulate and multiply in both directions. The ring's own redundancy mechanism - whether RSTP or a vendor ring protocol - cannot block a segment it does not manage, and the resulting frame rate saturates every link and overwhelms the PLCs' Ethernet interfaces. Telemetry is lost not because the fibre is cut but because the CPU and network interfaces are starved of capacity. Recovery required physically removing the loop, which is why it took 25 minutes.

The secondary, and more serious, cause: there was no local fallback. Sending pump start decisions from a central SCADA server means the stations depend on communications for their primary control. A wet well overflow is the predictable consequence. Local level control with a float or level switch and a hard-wired high-level start should have kept the station safe with the network completely dead.

Underlying control deficiencies, mapped against IEC 62443:

• No zones and conduits - control, CCTV and corporate systems on one flat 192.168.1.0/24 network with no firewall, so there is no security level boundary anywhere.
• Shared administrator credentials across all 22 PLCs, so there is no accountability and no way to revoke one contractor's access.
• Internet-facing VPN with a single shared account and no MFA - an unmonitored remote entry into the most critical layer.
• No asset inventory, no network drawings kept current, no configuration management and no offline backups, so recovery depends on one person's memory.
• Unsupported operating system with no patching path, and no test rig to validate patches before they reach production.
• No access control at the cabinet: a contractor with a laptop and a spare port is treated as normal.
• No permit-to-work clause covering network connection, and no pre-connection check by the client's engineer.

Order by risk reduction per unit of effort, not by elegance.

Before any further work at any station (days):

1. Remove or disable every unused switch port and physically lock the cabinets; a cabinet key is the cheapest access control there is. Engineering.
2. Add a permit-to-work clause: no device connects to the control network without a written method statement, a port number and the client engineer present. Procedural.
3. Change all shared administrator passwords to unique per-device credentials, held in a vault; revoke the contractor's shared VPN account immediately. Both.
4. Enable storm control, BPDU guard and loop protection on every managed switch; replace the unmanaged switches at all 22 stations, since an unmanaged device cannot be protected. Engineering.

Short term (weeks to months):

1. Take and store offline, version-controlled backups of every PLC and RTU program, and prove one restore in a test rig. Engineering/procedural.
2. Implement local fallback logic at each station - a hard-wired or locally programmed high-level start independent of the network. This is the single change that would have prevented the overflow, so it arguably belongs at the top of the list; it is placed here only because it needs design and outage time. Engineering.
3. Restrict the internet-facing VPN to a vendor-specific, time-limited, MFA-protected jump host, with session logging. Engineering.

Medium term (the real fix):

1. Segment into zones and conduits per IEC 62443: safety/control at the lowest level, SCADA above it, a DMZ with a firewall (or a data diode for outbound-only telemetry) at the boundary, and corporate and CCTV completely outside. Define target security levels per zone.
2. Build a patch and test regime: an isolated replica of the SCADA and PLC configuration, patch validation, then a change window. Replace the unsupported SCADA host.
3. Add network monitoring that alarms on new MAC addresses, broadcast rate and unauthorised device connections.

Priority logic: items 1-4 are cheap, fast and remove the exact failure mode that occurred. Item 6 removes the consequence. Items 8-10 reduce the probability of the next, different, incident.

Responsibility is shared, but not equally, and the allocation should be spelled out rather than argued.

• The contractor owns the immediate act: connecting an unmanaged device to a live control network without a documented method statement, without checking the port, and without confirming the network topology. A competent contractor working on an operational site should assume nothing about spare ports and should ask.
• The client owns the conditions that made the act catastrophic: undocumented topology, unmanaged switches, no local fallback, shared passwords, no permit clause and no supervision of the connection. Saying 'the drawings were wrong' is an admission of a configuration management failure, not a defence.
• Both parties failed the same fundamental test: neither treated the network as a safety-related asset requiring a change control process.

What the client's engineer should have done: issued a current network drawing and an asset list with the work order; specified the exact port and required the contractor to use it; required the contractor's laptop to be scanned and its IP/MAC registered; been present or nominated a competent person to be present during the connection; required a pre-work checklist covering what happens if the link goes down, including confirmation that local fallback exists.

Reporting:

• Raise an internal incident report the same day, including the timeline, affected stations, volume discharged and the fact that the overflow occurred.
• Notify the plant owner and, through the owner's environmental permit and incident notification procedure, the relevant authority for the discharge - an overflow is a reportable event and must not be handled as an internal IT matter.
• Record it in the safety and security incident register so that it feeds the next risk assessment and the IEC 62443 security level review.
• If the site is covered by a safety management system, raise a corrective action with an owner and a due date, and verify it closed.
• Report the near-miss in the industry sense: nine stations lost for 25 minutes is a near-miss for a much larger event, and lessons should be shared across the organisation's other sites.