CCNA: A Nightly Blocklist Job Across 200 Branch Routers

A Nightly Blocklist Job Across 200 Branch Routers

Your team pushes a nightly threat-intel blocklist to 200 IOS-XE branch routers. A Python script using Netmiko loops over the inventory and, for each of about 60 flagged addresses, sends access-list 101 deny ip host <addr> any, then disconnects. ACL 101 already contains access-list 101 permit ip 10.0.0.0 0.255.255.255 any followed by access-list 101 permit ip any any. Last week an address on the list stayed reachable for 36 hours even though the job printed done for every device. ACL 101 is now several hundred lines long and nobody can audit it. On a typical run 12 routers raise NetmikoTimeoutException; the loop wraps everything in except: pass. Credentials are hard-coded in the script, which lives in a shared repository.

1. The deny lines were present on the router, yet the address was still reachable. Explain the failure and how you would confirm it in 30 seconds.

Read the worked answer

ACL ordering. Global access-list 101 ... statements append to the end of the ACL, and the first matching statement wins. A deny that lands after permit ip any any is dead code: the config looks correct, the script exits cleanly, and the traffic is still permitted.

show ip access-lists 101     ! read top-down, find the deny relative to the permits
show ip interfaces           ! is 101 applied at all, and in which direction

Two failure modes to check together: placement (the deny sits below the blanket permit) and binding (the ACL is not applied to the interface, or is applied out when the traffic is inbound). A third, rarer one: the traffic is being permitted by a completely different ACL on another interface.

Fix: reserve sequence space so inserts land above the permits.

ip access-list extended 101
 10 deny ip host 203.0.113.9 any
 100 permit ip 10.0.0.0 0.255.255.255 any
 110 permit ip any any

Denies get 10-99, permits start at 100. Better still, put the volatile blocklist in its own ACL or object-group referenced before the base policy, so it can be replaced wholesale without renumbering the stable part of the ruleset.

2. Diagnose the 12 timeouts and explain what `except: pass` costs you on a job like this. What is the minimum change that makes the job trustworthy?

Read the worked answer

Likely causes of the timeouts, in order:

The account is privilege 1 on those devices, or no enable secret is supplied, so the session sits at > and configure terminal never opens the config prompt. Netmiko then waits for a prompt that will not arrive and raises a timeout.
Prompt-matching problems: an unusual MOTD/login banner, a confirmation prompt, or a device slower than the default read timeout.
Genuine reachability, AAA, or ACL problems on those 12 devices.

Do not guess - reproduce on one device with netmiko.ConnectHandler(..., session_log='dev.log') and read the transcript. That tells you in one run whether it is privilege, prompt, or path.

What except: pass costs:

You cannot tell which devices were skipped, so a 188-of-200 run is indistinguishable from a full run. That is exactly how a flagged address stayed reachable for 36 hours.
A failure in the middle of send_config_set leaves a device half-configured and possibly still in configuration mode.
You throw away the exception text you need to classify the failures.

Minimum change:

Per-device try/except that records the device name and exception text, then continues.
Call enable() with the enable secret, or use a privilege-15 service account - a policy decision, not a workaround.
A verification phase that re-reads the device and asserts the intended entries exist and are correctly ordered.
A PASS/FAIL/SKIP summary with a retry list, and alerting on failures instead of printing done.
Tune read_timeout / fast_cli for the slow models before blaming the network.

3. What would you change structurally? Describe the target design and how you would roll it out to 200 devices.

Read the worked answer

Replace 60 appends with one declarative state push.

Source of truth: the blocklist lives in Git and is rendered into the device's intended ACL by a template. The device config becomes a function of version-controlled intent.
Push with a diffing tool: Ansible cisco.ios.ios_config with parents: ip access-list extended BLOCKLIST and lines: only pushes statements that are missing, so a compliant device is a no-op. For true replacement, NETCONF <edit-config> or RESTCONF with default-operation="replace" on the ACL subtree makes the device's ACL exactly what the template says - stale entries disappear.

Why this matters: send_config_set is a line-by-line paste with no concept of desired state. It cannot remove entries, cannot guarantee order, and cannot tell you whether the device was already correct. A diff-based push is idempotent by construction.

Secrets: move credentials to Ansible Vault or a secrets manager, use per-device keys or TACACS+ accounts, and purge the repository history.
Rollout: lab device, then a 5-router canary, then 25, then the rest, with --check --diff before each step and a post-change verification that reads back the ACL and asserts the denies sit above the permits.
Scheduling: run from a scheduler with per-device results and alerting. 200 devices is also a natural fit for a controller-driven push, but the core change is the same: declarative, versioned, verifiable intent instead of 60 nightly writes.