Firmware Rollout Across 120 Substations Without an OT Test Lab
You are the lead SCADA security manager for an electric distribution utility. A vendor has disclosed a vulnerability in the communications module firmware of the RTUs at 120 substations. Exploitation requires access to the substation LAN and can force an RTU reboot, dropping remote control for 8 to 15 minutes. The vendor has released firmware, but it was validated only in the vendor lab; your own OT test lab was decommissioned last year. Each substation has redundant RTUs, but switching to manual control takes about 20 minutes per site. Field crews can visit 8 sites per day. The vendor offers a remote upgrade through a cellular modem, but that modem terminates on the same flat VLAN as the RTU. The summer peak change freeze starts in 21 days. The CISO wants a decision in 48 hours and asks whether to authorize remote upgrades for all 120 sites.
1. What options would you present to the CISO, and what criteria decide among them?
Read the worked answer
Decision frame. I would not present a binary 'remote versus defer.' I would present three options with explicit conditions:
- Remote mass upgrade - fastest, but rejected as the primary path. The cellular modem shares a flat VLAN with the RTU, there is no independent lab validation, and a failed upgrade across 120 sites could coincide with peak load.
- Staged field upgrade - crews at 8 sites/day, prioritized by load criticality and exposure. With 21 days before freeze that is at most roughly 168 site-visits, but travel and manual switching reduce realistic throughput to 40–60 sites. Do the highest-consequence sites first.
- Defer with compensating controls - only for sites not reached before freeze.
Criteria: exploit precondition (LAN access), safety consequence of reboot, restore time, rollback method, firmware hash verification, and regulatory evidence. I would authorize staged field upgrades plus interim controls: restrict cellular inbound to a vendor jump host, disable unused services, and monitor RTU reboot and communications anomalies. I would not authorize unsupported remote mass upgrade without a rollback test and a segmented upgrade path.
2. Define the compensating controls you would require for any site that remains unpatched at the change freeze, including evidence.
Read the worked answer
Controls must be tied to the actual exploit path, not a generic checklist.
- Network path control: deny inbound cellular access to the RTU except from a dedicated vendor jump host, and allow only the vendor upgrade port/protocol. Evidence: firewall rule export with change ticket and a packet-capture test showing blocked direct RTU access.
- Service minimization: disable unused RTU services and change default credentials. Evidence: configuration backup diff per site.
- Monitoring: alert on RTU reboot, unexpected configuration writes, and cellular session anomalies. Evidence: SIEM rule IDs and a 7-day baseline showing alert tuning.
- Manual operations readiness: verify each unpatched site has a current switching procedure and trained operator. Evidence: signed checklist per site.
- Vendor supervision: no remote vendor session without an authenticated, logged escort. Evidence: session logs retained for the audit period.
Each control gets an owner, test date, and review date in the risk register. Residual risk is recorded per site, not averaged across the fleet, because a single high-consequence substation can dominate the risk.
3. How do you handle the 21-day freeze and the regulator while still meeting the CISO's 48-hour deadline?
Read the worked answer
I would give the CISO a time-bound decision memo within 48 hours, then execute a risk-based exception.
Capacity math: 8 sites/day × 21 days is 168 visits, but realistic throughput is 40–60 after travel and switching. I would tier sites by load served, remote-control dependency, and exposure. The top tier gets field upgrades before freeze; the remainder gets compensating controls and a scheduled post-freeze upgrade.
Freeze exception: request a narrow change exception for security remediation, not a blanket freeze waiver. The exception lists sites, dates, rollback steps, and a stop-work condition if an upgrade fails twice.
Regulator: do not wait for the freeze to end and do not ask forgiveness. Provide a written plan showing the vulnerability, affected population, completed and pending sites, interim controls, and the residual-risk owner. If the regulator requires a risk assessment before controls are documented, use the same memo as the assessment record.
This is defensible because it prioritizes safety-critical sites, preserves evidence, and avoids an uncontrolled mass remote change under peak conditions.