Stale SCADA Data and a Remote Breaker Command
At a 220/33 kV substation, the SCADA RTU suddenly shows frozen values for 12 minutes: feeder currents, transformer tap position and breaker status all remain at pre-fault values, while local control panel indications are normal. The substation has redundant fibre links to the control centre, but link A failed last week and link B is now showing intermittent packet loss. The GPS time-sync source has also lost lock, and the RTU event log timestamps are inconsistent. A planned firmware patch for the RTU was deferred because the vendor could not confirm rollback. The control centre operator is about to issue a remote breaker-open command on a 33 kV feeder that has an earth fault and supplies a 2 MVA load including a hospital. The JE is asked to decide remote versus local operation and the restoration sequence.
1. Decide whether the remote open command should be executed and state the immediate control philosophy.
Read the worked answer
The remote command should not be executed from the control centre while the RTU data is frozen and the communication link is unreliable. The operator cannot trust the displayed breaker status, current or tap position, and a command sent over a link with packet loss may be delayed, duplicated or lost. Opening the wrong breaker or failing to open the faulted feeder would prolong an earth fault and endanger the public.
The immediate control philosophy should be local control with positive verification:
- Instruct the substation attendant to verify the faulted feeder locally by panel indication and, if available, a local current measurement.
- Open the correct 33 kV breaker from the local control panel, then confirm the breaker position mechanically and electrically.
- Inform the control centre immediately by telephone or radio and block remote commands to that bay.
- Coordinate with the hospital load, but do not delay isolation of a faulted feeder for load reasons.
- Record the time and the discrepancy between local and remote indications; this becomes evidence for the fault investigation.
Remote operation can resume only after the RTU data is proven fresh and the communication path is stable.
2. Give a structured troubleshooting sequence for the RTU, fibre and time-sync faults, including how to avoid making the system less reliable.
Read the worked answer
I would troubleshoot in a sequence that preserves at least one working path and one source of local indication.
- Confirm RTU power, CPU health, communication-card status and local HMI data. Compare local values with the control-centre display to confirm stale data.
- Check link B optical power, error counters and patch cords. Use a loopback test at the RTU end and then at the control-centre end. Do not disturb both links at once.
- Inspect link A and plan its repair, because losing redundancy is the main system risk.
- Check the GPS antenna, cable, satellite status and time-sync distribution. If GPS lock cannot be restored quickly, use a temporary NTP source, but label timestamps as suspect.
- Preserve logs before rebooting anything. Change one variable at a time, and avoid rebooting the RTU during an active fault unless local control is fully available.
- After restoration, compare every analogue and status point with local indications and test a command on an unloaded or isolated feeder.
The guiding rule is to keep local control and manual recording available until remote indication, command and time-stamping are all trustworthy.
3. Assess the deferred firmware patch and the change-management risk. What would you do before applying it?
Read the worked answer
The deferred patch should not be applied during the incident. A firmware change on an RTU with an unconfirmed rollback can turn a communication problem into a loss of control or a longer outage. The patch may correct the communication defect, but that is a hypothesis, not a proven cause.
Before applying it I would require:
- vendor confirmation of rollback procedure and a tested backup of the RTU configuration;
- a lab or spare-RTU test of the patch with the same communication cards and protocol settings;
- a formal change approval, maintenance window and named person in charge;
- a fallback plan using local control and manual logging;
- a cyber review of the logs, because unexplained freezing and packet loss can also indicate unauthorised access or a network fault.
If the vendor cannot support rollback, the patch should remain deferred until the communication fault is diagnosed and redundancy is restored. Applying an unproven patch to a critical substation RTU is not an acceptable way to save troubleshooting time.