Replacing CLI Screen-Scraping Automation on 300 Branch Routers
An operations team manages 300 Cisco IOS-XE branch routers running a mix of 16.12 and 17.3. Their change process is a Python script that opens an SSH session with Paramiko, sends configuration line by line, and waits for the prompt string. Last month the script pushed a new VTY ACL. On 12 devices an extra line applied successfully but the prompt regex did not match, the script timed out, and the session was left open. On six of those routers the ACL removed the NOC management subnet and engineers had to travel to site with a console cable. The team also cannot prove what changed: the only record is the script's stdout log.
The new requirement is idempotent, verifiable configuration of VTY ACLs and NTP servers across all 300 routers, with automatic rollback if a change breaks management access. All routers are reachable in-band only, and there is no console server.
1. Why is the current approach unsafe, and what should replace it? Include the device-side prerequisites.
Read the worked answer
Screen-scraping treats a structured device API as a terminal. The script makes decisions from prompt strings and free-text output, which vary by IOS-XE release, by terminal length, by enabled logging, and by the exact CLI parser behavior. It also never checks for error markers such as % Invalid input or % Incomplete command, so a partially applied change looks like success. There is no transactional boundary: line by line means any failure leaves the device in a state that was never intended and was never recorded.
Replace it with a model-driven interface. NETCONF over SSH port 830, or RESTCONF over HTTPS 443 if the team prefers JSON and HTTP semantics, returns structured <rpc-reply> or JSON bodies with per-request error tags, so a failure is detected as data rather than guessed from text. Configuration is expressed against YANG models such as Cisco-IOS-XE-native, ietf-interfaces and the ACL and NTP models, which also validates types and ranges before the device ever parses the CLI.
Device-side prerequisites on IOS-XE: netconf-yang (and restconf plus ip http secure-server for RESTCONF), aaa new-model with a user authorized to privilege 15, ip ssh version 2, and the appropriate aaa authorization for the NETCONF session. Verify the capability set from the NETCONF hello before writing code; older 16.12 images may not advertise the candidate datastore.
2. How do you make the VTY ACL change idempotent and automatically recoverable on a router reachable only in-band?
Read the worked answer
Idempotency comes from comparing intent with reality. Read the current configuration with get-config on the running datastore, build the desired YANG subtree for the ACL and the VTY access-class, and compute a diff. If the device already matches, do nothing; that removes the repeated-apply failures and makes the run safe to repeat. When a change is needed, send edit-config with explicit operation attributes, using merge for additions and delete or replace for entries that must be removed, because nested merge semantics can otherwise leave stale ACEs behind.
Recoverability is the harder half. If the device advertises the candidate datastore and the confirmed-commit capability, load the change into the candidate, run <validate>, then commit with a confirmed timeout of five minutes. If the automation does not send the confirming commit within that window, the device reverts to the previous configuration automatically. Run the reachability check from an independent probe during the window, then confirm.
If the candidate datastore is not available on the 16.12 images, fall back to the CLI safety net: enter configuration with configure terminal revert timer 5, let the automation apply and verify, and let the timer expire unless the change is explicitly confirmed. Either way, never apply a VTY change to all 300 devices at once; canary a small group first.
3. What does the team need in order to prove after the fact what changed and to detect unauthorized drift?
Read the worked answer
Three layers of evidence are needed, and none of them is a terminal log.
First, capture the pre-change and post-change configuration as data. Store the get-config reply for the relevant YANG subtrees, plus a full show running-config snapshot, in version control keyed by device and timestamp. Because both are structured, diffs are meaningful and reviewable, and they can be attached to the change ticket automatically.
Second, keep the transaction record: for each device, the RPC that was sent, the reply, the commit identifier, and whether the change was confirmed or reverted. That record proves intent as well as result, which stdout logs cannot do.
Third, detect drift continuously rather than on request. Poll the same subtrees on a schedule and alert when the live configuration differs from the intended state, or move to model-driven telemetry with a dial-out subscription so the device pushes configuration and oper-state changes as they happen instead of waiting for the next poll. Add syslog and TACACS+ command accounting for out-of-band CLI changes so a manual fix by an engineer is attributable, and make sure the drift check itself runs with read-only credentials so it can never cause the incident it is reporting.