A Flat Control Network, a Dual-Homed Historian, and 40 Seconds of Timestamp Drift

Zones and conduits (IEC 62443 approach)

Define zones by function and required security level, then connect them only through controlled conduits:

• Basic Process Control zone - DCS controllers, their engineering stations, operator HMIs.
• Packaging / PLC zone - the packaging PLC and its HMIs.
• Safety zone - any SIS, isolated from both.
• Plant DMZ - historian collector, patch/AV management server, jump host, remote-access gateway.
• Enterprise zone - corporate LAN.

Conduits: stateful industrial firewalls between zones, with explicit rules per protocol, port and direction, default deny, and logging. The historian collector is reached from the DMZ on one port only; corporate has no route into the control zones.

Why VLANs alone would not have helped

• A VLAN is a Layer-2 separation on shared switch hardware. There is no traffic inspection and no policy - the traffic still shares the same broadcast and spanning-tree domain.
• Common failure modes cross VLANs anyway: a misconfigured trunk, a rogue device, VLAN hopping, a misconfigured NIC team, or a spanning-tree loop taking down the whole switch stack.
• The 11-minute outage is the evidence: an enterprise scan generating broadcast/multicast traffic reached the process switches. That means there was no Layer-3 boundary and no filtering anywhere. A VLAN would have kept the frames off, but a single misconfigured uplink would have restored the exposure and you would never have known.
• Real separation needs routed Layer-3 boundaries plus a firewall, and for the highest-consequence zones, physically separate switches.

On the corporate AV/update demand: apply it in the DMZ. A WSUS/patch server and AV management server live in the DMZ; plant HMIs and servers are patched on a vendor-approved, tested, scheduled basis from there. Never push from enterprise to controller or HMI directly.

A dual-homed host is a router. Windows will route (or at minimum bridge at Layer 2 via bridging/ICS behaviour) between the two NICs, so the historian is an uncontrolled, unmanaged, unfiltered connection between the corporate LAN and the process network. It does two things:

• It propagates the corporate broadcast domain into the process network - which is exactly how a vulnerability scan produced a broadcast storm that reached the DCS switches and forced controller failover. Symptom (2) has a name: the historian.
• It gives every corporate host a potential route to control devices, and it gives every process device a route outward. Any perimeter you draw is fictional while it exists.

Data gaps (symptom 1): the 5–10 s gaps every 15 minutes line up with the backup job. The backup is saturating the historian's link or CPU at the same time as the collection job, and the collection queue overflows. Confirm with interface counters and queue drops on the corporate NIC during the backup window, and with the historian's collection diagnostics.

Replacement architecture

• Remove the corporate NIC entirely. Place the historian (or, better, a dedicated collector/interface node) in the plant DMZ with a single interface.
• Corporate consumers read a replicated copy in the DMZ. Collection from the DCS crosses the firewall in one direction only: DMZ-initiated to the DCS on a single documented port.
• Where the requirement is genuinely outbound-only (historian to enterprise), use a data diode or one-way gateway - it is the only design that survives a compromised corporate host.
• Move corporate backups off the collection window regardless, and put QoS or a dedicated interface on any remaining shared path.
• After the change, verify the path is not silently re-routing and that the corporate side can no longer reach a controller.

Diagnosis. A 40-second offset is not drift - it is a device free-running on its own clock, or different devices syncing to different sources (or not at all). Controllers, HMIs and the packaging PLC are evidently on independent clocks.

The requirement drives the method. Ask what the timestamps are for:

• Trend correlation across systems (minutes-scale questions): 100 ms accuracy is plenty. NTP to a local time server is adequate and cheap.
• Sequence of events (which tripped first, in what order, milliseconds apart): you need ≤1 ms, and often better. NTP over Ethernet typically gives 1–10 ms at best and degrades under load; you need IRIG-B or IEEE 1588 PTP with hardware timestamping. If the existing controllers support neither, that gap must be documented as a limitation, not papered over.

Design

• A GPS-referenced NTP/PTP source, with a local time server in the control zone that itself syncs outward through an explicit firewall rule (UDP 123). Do not open the corporate NTP servers directly to controllers.
• Configure time in the controllers and PLCs, not just the HMIs. The HMI is a display; the event record is in the controller.
• Set devices to UTC and do the timezone/DST conversion at the HMI layer. Half of all 'the timestamps are wrong' investigations end here.
• Verify: pull the current offset from every device after the change and keep the time-sync design in the network documentation, because the next person to add a switch will otherwise break it.