A Nightly Ansible Run Leaves 60 of 180 Switches Half-Configured

Freeze first. Disable the scheduled job and any manual runs so the state stops changing while you are measuring it.

Establish the blast radius. The playbook is the record of what was attempted; the running config is the record of what happened. For each of the 180 devices, capture show vlan brief, show ip access-lists and show running-config | include <changed markers>, then compare against the intended render. Sort devices into: untouched, VLAN-only, VLAN plus duplicated ACL, and failed/unknown. Do not trust the task output alone - check whether the failed task on switch-062 stopped the play for the remaining hosts or continued.

Do not restore from Sunday. A six-day-old backup would roll back legitimate changes made by other teams, may not reflect the current CMDB, and is a much larger, less reversible action than fixing the delta. It is the last resort, and only for a device whose running config is unreadable or unreachable.

Recover in order of safety. For each class, write a targeted corrective task that is idempotent: create or update the VLAN with the intended name, and for the ACL use ios_config with parents: (or the ios_acls resource module) so it replaces rather than appends. Run with --check --diff --limit on one device per class, review the diff, then apply with serial: 5 and max_fail_percentage: 0 so a failure halts the batch. Re-verify with the same show commands and only then re-enable the schedule.

The problem is not Ansible; it is the absence of pre-flight validation, batching and verification.

Playbook correctness. Make every task declarative: use resource modules (ios_vlans, ios_acls) or ios_config with parents: and match: so repeated runs converge instead of appending. Render the intended configuration from a template once per run and compare it to the device before applying. Fail explicitly if an existing VLAN ID has a different name in the intent source, rather than letting the device decide.

Pipeline gates.

• Lint and syntax check (yamllint, ansible-lint), plus a unit test of the rendered intent against a fixture.
• --check --diff in CI against a lab or snapshot, with a human reviewing the diff for the canary.
• NetBox as the inventory source of truth, with a test that every intended VLAN ID is unique and consistent across sites before deployment.

Execution controls. Always serial (e.g. 10) and max_fail_percentage, never serial: 0 for production. Canary one switch per site, then one site, then the rest. Pull a timestamped show running-config per device at the start of the run and store it in Git - weekly backups are useless inside a change window. Add post-change assertions: VLAN present with the right name, ACL present once, SSH and SNMP reachable, and a synthetic ping to the management gateway. If an assertion fails, stop the batch and run a corrective play automatically.

Rollback. Keep a rollback play tied to the run's own backup, not to the weekly archive, and rehearse it before it is needed.

Detect.

• A nightly no-change run with --check --diff, writing the diff per device into a Git repository. Any non-empty diff is drift, and the commit log becomes the audit trail of both intended and actual state.
• Compare running config against NetBox intent weekly, not just against the previous run, so slow manual edits that match the last diff still surface.
• Alert on drift with a severity based on the device role: a drifted distribution switch is more urgent than an access closet.

Govern.

• Every change flows through the repository: pull request, review, CI checks, approved change ticket, then the pipeline. Manual console changes are permitted only for break-fix, with a follow-up to bring the repository back in line within a defined window.
• Tag each device with a configuration version so the running state can be traced to a commit.
• Store the rendered intent, not just the playbook, so the expected output is reviewable by people who do not read Jinja.

Measure. Dashboard the count of drifted devices, failed runs, average time to remediate, and change volume per site. Review drift weekly and treat a rising trend as a process signal, not just a technical one.

Quarantine. Devices that repeatedly fail post-change verification should be excluded from automated runs and flagged for manual review, so the pipeline cannot keep reapplying a change that the device rejects. Document the whole loop - inventory, intent, pipeline, verification, drift - in the runbook so the next on-call engineer does not discover it at 09:00 on a Tuesday.