Configuration Drift Across 150 Branch Switches
Adatum has 150 branch switches from three vendors. A recent audit found 22% of devices deviate from the standard baseline, including missing SNMP ACLs and wrong NTP servers. Changes are made by three engineers over SSH, and there is no source of truth. Sixty switches run only SSH/CLI; ninety support NETCONF/YANG. The team has Git, Ansible, Python, and a read-only SNMP monitoring system. The change window is Sunday 02:00-05:00, and any failed change must be rolled back within 15 minutes. The security team requires credentials to be vaulted and all changes logged to the SIEM. Design an automation workflow, choose the transport per device class, and explain how you would validate and roll back. What would you automate first and why?
1. Design the end-to-end automation workflow, including the source of truth, rendering, approval, and execution.
Read the worked answer
Start with a source of truth in Git: one YAML or JSON file per site and role, plus a device inventory that records vendor, management IP, transport, and model. Do not store credentials in Git. A CI pipeline validates the schema, checks that every device has an owner and a role, and renders intended configuration from templates. A pull request is the approval gate; a second engineer reviews the diff before merge. On merge, the pipeline creates a change ticket, snapshots the current running configuration, and runs a dry-run or diff against the device. Execution is staged: canary switches first, then a wave of 10%, then the rest. Ansible can orchestrate both CLI and NETCONF devices and write structured results back to the ticket. The pipeline posts start, success, and failure events to the SIEM, and all credentials come from a vault at runtime. This workflow separates intent, approval, and execution so a bad template cannot reach all 150 switches at once.
2. Choose the transport for each device class and explain how you would validate and roll back within 15 minutes.
Read the worked answer
For the ninety NETCONF/YANG switches, use NETCONF with candidate datastore and confirmed commit where supported; that gives an atomic commit and an automatic rollback if the confirmation is not sent. For the sixty CLI-only switches, use Ansible network_cli with a saved pre-change configuration and a generated rollback script or configure replace. Validation should be protocol-level, not just 'the playbook finished': after the change, retrieve the configuration with NETCONF get-config or show running-config, parse the NTP servers and SNMP ACLs, verify NTP synchronization, and test that the management ACL still permits the monitoring subnet. Compare the rendered intent to the actual state and fail the wave if any canary differs. For rollback, keep a per-device backup and a tested restore path; on NETCONF use commit confirmed with a two-minute timer, and on CLI use the saved config with an automated restore playbook. A 15-minute rollback requires pre-staging the backup and testing the restore command, not writing it during the incident.
3. What would you automate first, and how would you handle the 22% drift without a risky big-bang remediation?
Read the worked answer
Automate the lowest-risk, highest-audit-value baseline first: NTP servers, SNMP ACLs, logging destinations, and banner or AAA settings. These are unlikely to break forwarding, are easy to validate, and directly address the audit findings. Run the pipeline in report-only mode across all 150 switches, produce a drift report, and group deviations by vendor and template bug. Fix the templates before touching devices. Then remediate canaries, then 10%, then the rest, with a freeze if any wave fails validation. Do not automate VLAN or routing changes until the baseline is stable, because those have larger blast radii. Also add a weekly compliance scan so drift is detected continuously, and make the Git merge the only path for standard changes. The goal is not only to fix the current 22% but to stop new drift from being introduced by manual SSH changes.