Configuration Drift Across 180 Branch Routers
A school district operates 180 branch routers managed by three engineers. Configurations are edited manually over SSH, and the only backup is a weekly TFTP copy to a NAS. An audit found 27 routers with an undocumented SNMP community string, 14 with a secondary default route pointing to a decommissioned ISP, and 6 with no NTP. The district has a Git repository, Ansible, and a Python-capable jump host, but no CI pipeline. A new CISO requires configuration changes to be reviewed, tested, and reversible, and the inventory to match reality. The routers come from two vendors; 12 are end-of-support but still in production. The team has 20 hours per week and a six-week deadline before a security assessment.
1. Sequence the remediation work and explain why a bulk push to all 180 routers in week one would be the wrong move.
Read the worked answer
Sequence from evidence to canary to waves:
- Week 1: reconcile inventory and back up every configuration before changing anything. Use SSH, SNMP, CDP/LLDP, and ARP data to build a source of truth with vendor, model, support status, management address, and site. Store configurations in Git with a hash. Freeze manual changes.
- Week 2: build vendor-specific templates for SNMPv3, NTP, and removal of the decommissioned static route. Test in a lab or on two lab routers. Define post-checks: management reachability, NTP offset, SNMPv3 authentication, routing adjacency, and no old static route.
- Week 3: canary six routers, three per vendor, using check mode and commit-confirmed or a scheduled rollback. Observe for 24 hours.
- Weeks 4–5: push in waves of 20–30, with a rollback checkpoint after each wave.
- Week 6: handle the 12 end-of-support devices manually, then validate inventory and evidence for the assessment.
A bulk push in week one is wrong because one bad management ACL or SNMP change can lock out 180 sites at once, and the weekly TFTP backup is too stale to recover quickly. The two vendors have different syntax and failure behavior, and 12 unsupported routers may not support the same automation path. The team has 20 hours per week; a failed 180-device push would consume the entire deadline.
2. How would you handle the 12 end-of-support routers and the two-vendor mix within the six-week deadline?
Read the worked answer
Do not force the 12 end-of-support routers through the same template as supported devices. Put them in a restricted management VLAN with an ACL that allows only the jump host, require console access for changes, and apply compensating controls: no internet-facing management, SNMPv3 where supported, and close monitoring. Document a replacement date. If a device cannot support SNMPv3 or NTP, record the exception and the risk rather than silently leaving the old community string.
For the two vendors, separate intent from syntax. Define a vendor-neutral data model for the required state: NTP servers, SNMPv3 user and access, management source, and no secondary default route to the decommissioned ISP. Render that intent into per-vendor templates and inventories. Run syntax and check mode per vendor, and never mix both vendors in the same canary wave until each template has passed independently.
Prioritize the audit findings: 27 undocumented SNMP strings first, 14 stale default routes second, 6 missing NTP third. The 12 unsupported devices are a separate workstream, not a reason to delay the 168 supported ones. With 20 hours per week, reserve at least four hours for rollback testing and two hours for evidence collection so the assessment does not become a last-minute fire drill.
3. Define the change pipeline, validation, rollback, and success metrics the CISO should see.
Read the worked answer
Pipeline: engineer creates a branch, opens a pull request, and a second engineer reviews it. CI runs YAML lint, Ansible syntax check, template rendering, and a diff against the running configuration. Merge triggers a canary job, not a full push. After canary post-checks pass, the wave job runs with a maximum failure threshold of one device. Secrets live in a vault, not Git; use SNMPv3 authPriv and remove community strings.
Validation after each device: management reachability from the jump host, NTP offset under 50 ms, SNMPv3 authentication success, routing adjacencies up, no stale default route, and expected configuration hash. Store results per device and fail the wave if any check fails.
Rollback: archive the pre-change configuration with a timestamp, use commit-confirmed or a scheduled reload where supported, and keep an out-of-band console path. If post-checks fail, restore the archive and open an incident. Test the rollback on a canary device before the first wave.
Success metrics: 100 percent of the 180 routers reconciled to the source of truth, zero undocumented SNMP community strings, 100 percent NTP compliance or documented exceptions, zero secondary defaults to the decommissioned ISP, 95 percent of changes going through the pipeline, drift detected within 24 hours, and mean time to restore under 30 minutes. Report these weekly to the CISO so progress is visible before the assessment.