Always-Changed Ansible Playbooks on 64 Junos Devices

cli_config with pre-built set commands is a command sender, not a state manager. It opens a configuration session, pushes the lines, and commits. It has no model of what the device currently looks like, so it cannot conclude "this is already correct." The commit succeeds, the device reports success, the task reports changed - every time, whether or not anything changed. The same applies to cli_command running show output that is never parsed into a comparable value.

For a dry run to mean anything, three things must hold:

1. Desired state is expressed declaratively per resource, not as ordered CLI strings. Use resource modules with a device model behind them (for Junos, the junos_* family, or junos_config with explicit lines plus a src/replace semantic that performs a diff). The module must read current state, compare, and only then act.
2. The comparison must be against a declared source of truth, so "correct" is defined by the repository, not by whatever the device happens to have. Partial management - where you push three lines and ignore the rest - cannot detect a reverted SNMP community, because the module was never asked to own that object.
3. The diff must be small enough to read. 700 changed tasks is the same as zero. Scope the nightly run per device role or per site, and emit a per-device diff, not a task count.

Until (1) and (2) are done, a nightly dry run only automates the production of noise.

What runs. A check-mode (--check --diff) pass over the declarative playbooks, executed per device group (core MX, aggregation ACX, access EX) rather than as one blast. Check mode must be genuinely non-committing - verify on one device first that a check run leaves no configuration session behind, since a stuck session will eventually block future commits.

When. Nightly for the full estate, plus on demand after any maintenance window, because the vendor-window example shows drift arrives through the front door, not through a rogue engineer. Also run immediately after every production apply, to prove the apply converged; a nonzero diff there is a pipeline failure, not a drift ticket.

Classification. Three buckets, and the taxonomy is the whole value of the loop:

• Expected/transient: counters, uptime, DHCP-derived values - excluded from comparison at the module level, not filtered out afterward by a human.
• Drift requiring remediation: a managed object differs from the repository. Route to a ticket with the exact diff attached and an owner by device role.
• Unmanaged change: a configuration element present on the device that no playbook claims. This is the category that would have caught the reverted SNMP community, and it needs a different response - either import it into the repository or remove it deliberately.

Remediation gating. Automated remediation only for objects classified as owned and non-disruptive, and only after the check output has been trustworthy for a few weeks. Everything else goes through human review with a rendered diff, then a controlled apply in a maintenance window. Track drift age as a metric: eleven days of unnoticed drift is the failure mode being fixed, so a drift ticket that sits open is itself an alert.

The absence of a lab is real, but "production at 02:00" is not the cheapest available option - it is the most expensive one, because every mistake is a customer-visible outage at the worst hour for the people fixing it.

Cheap confidence, in increasing cost:

• Syntax and policy gates before touching anything: commit check semantics, plus a lint/dry-render step in CI that produces the exact config diff for review. A diff a human reads before 02:00 catches far more than a diff nobody reads after.
• Check mode against production read-only as the pre-flight: if a playbook that should be idempotent reports changes on an untouched device, the playbook is wrong and you learn it before the window.
• Canary subset: apply to two low-impact devices first, verify with an automated post-check (interfaces up, BGP sessions established, expected routes present), then proceed. This is not a test environment but it bounds blast radius.
• Commit confirmed with a rollback timer: apply with commit confirmed, let the change run for a bounded interval, and require an explicit confirm from automation that passes the post-check. If the confirm never arrives, the device reverts itself. This is the single highest-value change given no lab.
• A virtualized lab is not out of reach: containerized or VM Junos instances can validate config syntax and most routing policy without hardware. Even two instances cover the highest-frequency mistakes.

Sequence: commit-confirmed plus post-checks first (no new infrastructure, immediate risk reduction), then virtualized validation, then canary as standing practice.