Pump Station Telemetry Failure During a Peak Demand Event
A booster pump station with three 500 gpm vertical turbine pumps (two duty, one standby) feeds a pressure zone serving 9,000 customers. The station is controlled by a programmable logic controller that maintains discharge pressure at 65 psi and alternates lead and lag pumps. During a hot summer afternoon, the SCADA system loses communication with the station. The operator at the treatment plant sees the station as offline but cannot command it. The station's local controller continues to run, but the operator later learns that the lead pump tripped on overload and the lag pump did not start, so discharge pressure dropped to 42 psi for about 40 minutes before a manual reset. Customers at higher elevations reported low pressure, and a few reported discolored water. The station has a high-pressure alarm and a low-pressure alarm, but the low-pressure alarm was routed only to the SCADA screen that was offline. The utility's emergency response plan lists the station but does not specify a telemetry-failure procedure. The operator must decide what to fix first and how to prevent recurrence.
1. What is the immediate operational priority after restoring the station, and what should the operator verify before declaring the event over?
Read the worked answer
The immediate priority is to restore and verify service, then protect public health. After the manual reset, the operator should confirm discharge pressure is back to setpoint, both duty pumps are available, and the standby is functional. The operator should verify pressure at the far end and at high-elevation points, not just at the station, because the station gauge can look fine while the far end is still recovering.
The discolored water reports matter: a pressure drop can allow backflow or sediment disturbance, and low pressure can create a pathway for contamination. The operator should flush affected areas, check chlorine residual and turbidity at representative points, and consider a precautionary boil water notice if pressure fell below the threshold the state uses (commonly 20 psi) or if contamination is possible. Here pressure fell to 42 psi, above 20 psi, so a boil notice may not be automatically required, but the operator should document the pressure history and the flushing and sampling results before closing the event. The reasoning is that pressure recovery is not the same as water quality recovery.
2. Why did the lag pump fail to start, and what does the operator need to check in the control logic?
Read the worked answer
The lag pump not starting suggests the control logic did not recognize the lead pump failure, or the lag pump start was interlocked or timed in a way that prevented it. The operator should review the local controller program: whether a pump running status or overload trip is an input that triggers the lag pump, whether there is a lead/lag alternation that left the lag pump unavailable, and whether the lag pump had its own permissive or interlock that blocked start. The operator should also check the lag pump itself for a trip, breaker, or control power issue.
The reasoning is that a standby pump is only useful if the control system automatically brings it on when the duty pump fails. The operator should test the failure sequence: simulate a lead pump trip and confirm the lag pump starts within the intended time. Document the test and any program changes. This is a control-reliability issue, not just a pump issue.
3. How should the operator address the alarm routing and the missing telemetry-failure procedure?
Read the worked answer
The low-pressure alarm routed only to an offline SCADA screen is a single point of failure. The operator should recommend redundant alarm routing: local annunciation at the station, dial-out or text alarms to on-call staff, and an alarm at a location that remains visible when SCADA is down. The operator should also add a telemetry-failure alarm so the loss of communication itself is annunciated, not just visible as an offline status.
For the procedure, the operator should draft or request a telemetry-failure section in the emergency response plan: who is called, how the station is checked locally, how long the station can run unattended, and what triggers a boil water notice or public notification. The reasoning is that the event exposed two gaps, alarm visibility and a written response, and both are correctable. The operator should document the event, the corrective actions, and the test results, and share them with the utility's emergency planning lead.