Nightly Reboots at Traffic Intersection 42
Metro City's traffic management center operates 420 cellular-connected signal controllers. A vendor maintains the central ATMS and holds a persistent VPN into the controller VLAN. At 02:10 each night, the controller at intersection 42 reboots; the next morning a local timing plan appears that was not in any approved change window. The firmware hash matches the signed baseline, but the SOC sees repeated TLS sessions from that controller's subnet to an IP address never seen before. The city's incident response plan exists but has never been tested. Traffic engineering can place the intersection in flashing or local control, but doing so during rush hour would cause gridlock. The state DOT requires notification of “significant traffic system security incidents” under the funding agreement, and the city has a public-safety spokesperson.
1. Triage the intersection 42 evidence: compromise, malfunction, or insider? What immediate actions are defensible?
Read the worked answer
The evidence is ambiguous but cannot be treated as a malfunction. A nightly 02:10 reboot suggests a scheduled task or watchdog, but the unauthorized timing plan and new TLS destination raise compromise or insider action. A signed firmware hash only proves the stored image, not runtime integrity. Immediate actions: preserve volatile evidence if the controller supports it (running config, logs, active sessions), capture packets at the cell edge or a span port, and compare the timing plan against change records. Place the intersection in local control or flashing mode only with traffic engineering's approval and at a safe time; do not power down the controller. Restrict the controller's egress at the APN or central firewall to the approved ATMS endpoint, and disable the vendor VPN for that subnet. Notify the incident commander and legal per policy. The judgment is to contain the network path first while keeping signals safe.
2. How would you contain the incident without stopping traffic?
Read the worked answer
Containment must be traffic-safe and reversible. At the central ATMS, apply an allowlist so the controller can reach only the management endpoint and NTP; block the rare IP and any new destinations. On the cellular side, move the controller to an APN profile with the same allowlist. Disable the vendor's persistent VPN for the controller VLAN, not just the one device, and require brokered access for the investigation. Keep the controller running; if it must be isolated, coordinate with traffic engineering to put the intersection in local control and post officers only if warranted. Rotate credentials and certificates used by the controller and the vendor, starting with those the contractor could have seen. Monitor for reboot recurrence and for other controllers contacting the same IP. Document every action with timestamps because the state DOT notification will depend on the timeline.
3. What governance and notification steps follow, and what should the post-incident review fix?
Read the worked answer
Activate the untested incident response plan and assign roles: incident commander, traffic engineering liaison, communications, legal, and the vendor. Notify the state DOT under the funding agreement once you have a credible timeline; do not wait for full root cause, but distinguish confirmed facts from hypotheses. The spokesperson should speak to service impact, not technical speculation. Preserve evidence and chain of custody.
Post-incident, fix the governance gaps that made this hard: a tested IR plan with a traffic-specific playbook, an asset and certificate inventory, MFA and just-in-time access for the vendor, egress allowlists by default, and runtime integrity monitoring rather than firmware hash checks alone. Add a tabletop with traffic engineering and the vendor, and require the vendor to disclose an SBOM and its remote-access architecture. Track corrective actions to closure with an owner and date.