Brownfield Automation Pipeline for 1,400 Switches
An enterprise manages 1,400 Catalyst switches across 40 sites with Cisco Catalyst Center 2.3.7, Ansible, and Terraform. A recent Ansible playbook pushed a VLAN change to 300 access switches; 12 failed because their interface templates differed, and a rollback playbook re-applied the previous running configuration, overwriting manual fixes made by the local team. The team also hits Catalyst Center API rate limits during large jobs and occasionally double-onboards devices after a failed task retry. Change windows are four hours long, and the CAB requires evidence of pre-checks and a tested rollback. The network team wants a repeatable pipeline that can handle brownfield devices without taking down access ports.
1. Design the pipeline stages and pre-checks for a VLAN change across 300 brownfield switches.
Read the worked answer
Use a source of truth, such as NetBox, to render intent, then validate the schema before touching devices. Add pre-checks for configuration drift, interface template match, PoE budget, uplink redundancy, and a per-device configuration snapshot. Run a dry-run or diff against Catalyst Center, then a canary wave of five switches, then waves by site. The reasoning is that brownfield failures are usually mismatched templates or local exceptions, not the VLAN command itself. The CAB evidence should include the diff, canary results, and the exact rollback artifact. Idempotency and state locking prevent duplicate onboarding and concurrent runs.
2. How should the team handle Catalyst Center API rate limits, retries, and partial failure?
Read the worked answer
Treat rate limits as a design constraint. Use asynchronous task polling, batch by site, honor 429 Retry-After, and apply exponential backoff with jitter. Use deterministic resource names and check-before-create so retries do not double-onboard devices; store device UUIDs in the source of truth. For partial failure, keep a per-device transaction log, stop the wave when a failure threshold is crossed, and quarantine failed devices for review. The reasoning is that blind retries cause duplicates, while per-device state allows safe resumption without repeating successful changes.
3. The CAB asks for a tested rollback. Why was the previous rollback unsafe, and what should replace it?
Read the worked answer
The previous rollback re-applied the full running configuration, which overwrote manual fixes made after the snapshot. Replace it with a diff-based rollback: capture the pre-change configuration, compute the exact changed lines, and roll back only those lines with human approval. Validate post-rollback with the same pre-checks and a canary. Keep a configuration archive for audit. The reasoning is that brownfield devices have local exceptions, so a full restore is destructive; a scoped rollback preserves local state while reversing the intended change.