Intermittent Remote I/O Rack Fault on a Packaging Line
A packaging line uses a PLC with a local rack and two remote I/O racks connected by EtherNet/IP. Remote rack 2 faults intermittently every 4 to 8 hours; the fault clears after a power cycle or sometimes by itself. The line stops only when the fault lasts longer than the PLC's 500 ms connection timeout. The remote rack is 120 ft from the PLC, and its Cat 5e cable runs parallel to a 480 V feeder for about 40 ft. The rack's 24 VDC supply also powers eight solenoid valves, and the 120 VAC output module switches those valves without individual surge suppressors. The faults began after a new 40 hp VFD was installed near the cable route. Maintenance has replaced the Ethernet cable and the remote rack power supply once, with no lasting change. There is no managed switch; the PLC and both racks share an unmanaged switch with office printers and a wireless access point.
1. Separate the plausible causes into communication-path, power-quality, and configuration categories, and describe a test that would discriminate among them.
Read the worked answer
Split the evidence by mechanism:
- Communication path: the fault is a rack connection timeout, and the cable runs 40 ft parallel to a 480 V feeder. VFD switching noise can couple into the Ethernet cable, especially if the shield is not bonded or the cable is not industrial-rated. A test is to route a temporary shielded cable well away from the feeder, or run the rack on a short local cable, and see whether the fault interval changes. Also check switch statistics, CRC errors, and duplicate IP addresses.
- Power quality: the rack's 24 VDC supply also feeds eight solenoids. Solenoid de-energization produces inductive kickback; without suppressors, the 24 V rail can dip or spike, resetting the rack's power supply or I/O modules. A test is to monitor the 24 V rail with a scope or fast min/max logger at the rack terminals during valve operation, and to temporarily power the rack from a separate clean supply.
- Configuration and loading: the flat network with office printers and a wireless access point allows broadcast traffic and possible IP conflicts. The 500 ms timeout is not generous. A test is to place the PLC and remote racks on an isolated VLAN or managed switch and check the connection timeout and RPI settings. Also confirm the rack's power supply is sized for inrush and that the 120 VAC output module is not being asked to switch beyond its rating.
2. What corrective actions would you implement, and in what order, to make the fault stop recurring?
Read the worked answer
Implement in order from least invasive and most likely:
- Separate the 24 VDC supply: give the remote rack its own supply, or at least separate the solenoid loads with a dedicated supply and add flyback diodes or RC snubbers across every solenoid coil. Verify the 24 V rail stays within the module tolerance during valve actuation.
- Fix the network path: replace the parallel run with a route at least 12 in from the 480 V feeder, use shielded industrial Ethernet cable, bond the shield at both ends to the enclosure, and separate it from motor cables. If that is impossible, use fiber.
- Isolate the control network: put the PLC and remote racks on a managed switch with an isolated VLAN; remove the office printers and wireless access point from that segment. Set a suitable RPI and connection timeout; do not simply increase the timeout to hide the fault.
- Check the VFD installation: confirm the VFD input and output cables are shielded and grounded, and that the drive output cable does not run alongside the Ethernet cable.
- Verify rack power and grounding: check the 24 V supply loading, terminal torque, and the rack equipment grounding conductor. Replace the power supply only after the load and noise are addressed.
- Retest: run the line through several VFD acceleration cycles and log the result.
3. What records or monitoring would you put in place to prove the fix?
Read the worked answer
Put in place a simple but defensible record:
- Network logs: enable the managed switch port statistics and syslog, and log CRC errors, dropped packets, and link flaps with timestamps. Correlate them with VFD starts and solenoid operations.
- Power monitoring: install a 24 VDC logger or a scope with a trigger on the rack supply; record minimum and maximum voltage and transients during a shift. Keep the trend for at least two weeks.
- Fault log: have operators record the exact time, line state, and any recent valve or VFD operation when the rack fault occurs. Compare with the PLC fault log and the drive event log.
- Change record: document each change (supply, cable route, switch configuration, snubbers) and the fault frequency before and after. A fix is only proven if the mean time between faults increases beyond the previous 4–8 hour interval for a sustained period.
- Baseline: record the final RPI, timeout, IP addresses, and switch configuration so future technicians do not undo the separation.