Ansible Check Mode and Canary Rollback on NX-OS
A network automation team manages 24 Nexus switches across four sites with Ansible. The GitLab CI pipeline runs ansible-playbook -i inventory site.yml --check --diff on every merge request, then applies on main. site.yml uses cisco.nxos.nxos_config with lines, parents, and match: line; connection is network_cli, and credentials come from a CI variable. Check mode frequently reports changed on every run even when nothing changed, and one MR failed during check mode with a check mode not supported error. A previous production change to an interface description was applied to all 24 switches at once and caused a monitoring outage because the description string was used by an automation parser. The team wants canary deployment, idempotency tests, and a safe rollback, but cannot add a lab device for every model.
1. Explain why `--check` with `nxos_config` and `network_cli` is unreliable here, and propose pre-merge validation that does not depend on check mode.
Read the worked answer
--check with network_cli is not a reliable pre-merge test for NX-OS. Some modules do not implement check mode and fail instead of predicting; nxos_config may still connect, gather running config, and compare text, but ordering, defaults, and match: line can make an unchanged config appear changed. The diff is also generated from the device's current state, not from a reviewed intended state. Better validation: render the intended config with Jinja2 from inventory variables, then use nxos_config with diff_against: intended and intended_config (or a nxos_command show run plus a local diff) to compare. Run ansible-playbook twice in a controlled canary and require the second run to report changed=0. Add --syntax-check, yamllint, and a unit test for the rendered template. Keep --check only for modules known to support it, and fail the MR if the module returns not supported rather than ignoring it.
2. Design the pipeline stages, canary strategy, health checks, and rollback for the interface-description change.
Read the worked answer
Pipeline stages: (1) lint and render; (2) intend/diff against running config on a canary switch per site and model; (3) manual approval; (4) canary apply with serial: 1 and max_fail_percentage: 0; (5) health checks (interface state, monitoring parser, reachability); (6) remaining batches by site with serial and pauses. Do not apply to all 24 at once. Before apply, save a config backup or checkpoint per device; after apply, run the same health checks and compare. For rollback, use configure replace from the saved checkpoint or a generated reverse diff, and test rollback on the canary first. The interface description should be treated as a monitored contract: validate the exact string against the parser's expected pattern before deployment. If no lab device exists per model, use the least-critical production canary per model and a maintenance window, not an untested all-device push.
3. How should credentials, concurrency, and idempotency testing be handled so the pipeline is safe and repeatable?
Read the worked answer
Credentials: use Ansible Vault or a CI secret store, pass them as environment variables, and never commit them to inventory or group_vars. Use a dedicated automation account with least privilege and AAA command authorization. Concurrency: keep forks low and use serial so the pipeline does not open 24 simultaneous SSH sessions; NX-OS control-plane and AAA may throttle. Idempotency: after the first apply, rerun the same play and require changed=0; if not, inspect the module output and switch to a more precise match or fix the template ordering. Use register and failed_when to assert expected states rather than relying on color. Add a lock in CI per site so two pipelines cannot change the same devices. Log the rendered diff and the device checkpoint ID with the pipeline run for audit.