A Netmiko Script Leaves 22 Switches Half-Configured
A managed-services provider supports 60 Cisco Catalyst 9200 switches for one customer. Every Monday a junior engineer runs a Python script that SSHes to each device with Netmiko, enters configuration mode, types vlan 210, name <name>, then interface range Gi1/0/1-12 and switchport access vlan 210, and finishes with write memory. Last week the run died after 22 devices because one switch presented a banner after a recent upgrade, leaving 22 devices with VLAN 210 and 38 without; a re-run with an edited VLAN name produced inconsistent names and left some ports at the old PVID. The customer now asks for an audit trail, a way to prove configuration matches intent, and a rollback path. The team has Ansible 2.15 available, and the switches run IOS-XE 17.9 with NETCONF and RESTCONF enabled on a management VRF.
1. Explain why the current script is unsafe and non-idempotent, and list the concrete changes that would make a Monday run safe to repeat.
Read the worked answer
The script is a procedural screen-scraper: it has no notion of desired end state, no per-device transaction, and no error handling. Three things follow. It cannot tell 'already correct' from 'needs change', so re-runs re-apply commands and produce drift (renamed VLANs, ports left on an old PVID). It stops at the first prompt that does not match the regex, so 22 devices changed and 38 did not, and nothing was rolled back. And write memory per device means a partial run is permanent.
Concrete changes: (1) Drive from a structured inventory and data file (YAML/JSON) held in git, with the VLAN list and port map as the source of truth. (2) Make the operation declarative - Ansible cisco.ios.ios_vlans with a full desired list and purge behavior, plus ios_l2_interfaces for the port assignment, so the task converges rather than appends. (3) Run --check --diff in a dry-run window and review output before applying. (4) Add robust connection settings: terminal length 0, disable paging, handle banners and enable prompts, and set persistent_connect_timeout. (5) Apply per device with a rollback path - a config backup first, and on IOS-XE use configure replace or configure terminal revert timer 10 so an unreachable device reverts itself.
2. Compare the Ansible resource-module approach with a Python NETCONF/YANG approach for this fleet. Which would you choose for VLAN changes, and why?
Read the worked answer
Ansible resource modules (ios_vlans, ios_l2_interfaces) speak over network_cli and translate a desired data structure into the minimal CLI. They are idempotent, diff-friendly in check mode, and require no YANG modelling knowledge - a good fit for a 60-switch fleet and a small team already using Ansible ad hoc.
A Python NETCONF approach against Cisco-IOS-XE or IETF YANG models gives stronger guarantees: structured replies instead of parsed text, candidate datastore plus commit confirmed, validate, and per-transaction rollback. It scales better when you also want streaming telemetry and model-driven config. The cost is tooling: you must model the VLAN and interface hierarchies yourself (ncclient plus lxml, or a library), manage namespaces, and handle IOS-XE model revisions.
Recommendation: use the Ansible resource modules for the weekly VLAN/port task, because the team can maintain it and the diff output is auditable. Use NETCONF with commit confirmed for high-risk changes such as trunk or management-plane edits, where the auto-revert is worth the complexity. Do not mix ios_config with raw lines for VLANs: that path re-applies text and gives you neither idempotency nor a reliable diff.
3. Design a drift-detection and audit process that proves the 60 switches match intent, and name the failure modes it must catch.
Read the worked answer
Keep the intent in git: a YAML file per site listing VLANs, port assignments, and trunk allowed lists, with a CI pipeline that validates syntax and runs ansible-playbook --check --diff nightly against all 60 devices, failing the job if any device reports changes. That turns drift into a ticket rather than a surprise.
For an audit trail, archive the running config before and after each change to a timestamped git commit or a config-archive server, and tag the commit with the change ticket. On IOS-XE you can also use the built-in configuration archive with time-period for periodic snapshots.
For deep verification, poll RESTCONF/NETCONF GET on the VLAN and interface YANG models and compare structured output against the source of truth - this catches things CLI diff misses, such as a PVID that matches a name but not the intended VLAN, or a trunk carrying an extra VLAN.
Failure modes to catch explicitly: a device added to inventory but never configured, a device with an unexpected banner or prompt that breaks screen scraping, a manual engineer change, a failed mid-run state, and a VLAN name that changed between runs. Alert on check-mode diffs and on NETCONF/RESTCONF unreachable hosts.