A Netmiko Push That Was Not Idempotent

send_config_set is a line-by-line CLI injector. It has no idea what is already on the device. ip access-list extended MGMT followed by permit tcp any host 10.0.0.5 eq 22 appends another identical ACE every run - the device does not deduplicate and, with no sequence numbers, there is no key to match on. That is the duplicate. The same property explains the lockout: the ACL is applied by a separate line, and when the session died mid-block the device kept whatever lines had already been accepted. Netmiko has no transaction, so there is no rollback and no all-or-nothing semantics.

The three things that actually fix this:

1. Model the object, not the lines. Either use an idempotent module (Ansible cisco.ios.ios_acls, or RESTCONF against the ACL container) or, if you stay with CLI, always emit explicit sequence numbers so a rerun replaces ACEs by key instead of appending:

10 permit tcp any host 10.0.0.5 eq 22
20 deny ip any any log

1. Make the push transactional. Use NETCONF with the candidate datastore plus confirmed-commit (or configure replace with a full rendered config) so a lost session leaves the device untouched, and a bad change self-reverts when nobody confirms it.
2. Never let the change sever its own transport. Order the push so the ACL is fully built and verified before it is attached, and require an out-of-band path (console server, or a permit for a management subnet that is already present) before applying anything that touches VTY access.

The general principle: if the script has to know the current config to decide what to send, it is not describing state, it is performing surgery.

Not necessarily - a rendered template is authoritative over the fields it owns, and a description in the YAML is a field it owns. The script did what it was told. The failure is that nobody had defined the boundary between managed and unmanaged configuration, so an out-of-band change was indistinguishable from drift.

What you need is an explicit ownership contract:

• Owned fields are rendered from YAML and reconciled on every run. Drift on these is an error to be corrected.
• Unmanaged fields are never touched - carve the WAN interfaces out of the template entirely if the network team manages them by change control.
• Drift on owned fields should be reported before it is corrected, so a deliberate emergency change becomes a conversation rather than a surprise revert. Practically: run the tool in a diff/check mode on a schedule, alert on non-empty diffs, and only auto-remediate what is in the owned set.

A rewrite of the WAN description in the YAML by the on-call engineer, committed through the same pipeline, is the process fix - it makes the emergency change visible in the source of truth, so the next run reinforces it instead of undoing it.

The judgement call is where to draw the line. Making everything owned gives you convergence and surprises; making nothing owned gives you a script that does nothing useful. Per-interface ownership with a diff report is the workable middle.

Do not guess supported features from the version string - ask the device. Two mechanisms get you this:

1. Capability discovery at run time. The NETCONF hello advertises supported YANG modules and revisions, and ietf-netconf-monitoring lets you pull the schema for a specific module. Where the platform supports the model but at an older revision, get-config on the container will show which leaves actually exist. Where the model is absent entirely, the device cannot be managed that way and must fall back.
2. A small set of versioned profiles, not 60 templates. Group devices by capability, not by hostname: one profile for the modern releases, one for the legacy ring, with the divergent snippets isolated as profile-specific fragments and the shared 90 percent common. A host_vars/inventory attribute selecting the profile keeps the diff between them reviewable.

Then gate the push:

• Run the pipeline against the legacy ring first as a canary. A platform/model mismatch shows up on two switches, not sixty.
• Fail closed: if capability discovery cannot confirm that every leaf in the rendered payload is supported, the run should refuse to push rather than send a partial config.
• Log the discovered module revision per device at every run. That gives you the date and cohort when a firmware upgrade changes behaviour, which is otherwise very hard to reconstruct after a bad rollout.

The point of the exercise is not to support both versions forever - it is to make the version split visible and bounded, so the upgrade to a single release is a scheduled project rather than an emergency.