Four-eyes change control and five-minute rollback for 240 switches
A managed-services provider operates 240 AOS-CX switches across six data centres, managed today by ad-hoc Ansible playbooks and a shared enable password kept in a wiki. Change control requires a second pair of eyes; the audit team wants proof that a bad change was rolled back within five minutes; and the last outage traced to a hand-edited VLAN change on one leaf that nobody replicated. The team wants to move to a pipeline with a source of truth. They already run NetBox, Git, a CI runner and TACACS+, and have roughly 15 device models and three different AOS-CX releases in service.
1. Design the pipeline stages and justify what the pipeline should and should not be allowed to write.
Read the worked answer
Stages:
- Render intent from NetBox (devices, interfaces, IPs, VLANs, VRFs) through templates. Render fragments, not a full replacement config.
- Diff the rendered intent against the live running config read via the REST API, using an order-insensitive semantic diff. A noisy text diff trains reviewers to ignore it.
- Validate on a canary in each site and each of the three AOS-CX releases - three releases means three template variants; treat version skew as a first-class input, not an exception.
- Approve via PR: the reviewer's approval plus the CI identity in the commit is the four-eyes evidence.
- Apply, then post-check: routing/EVPN neighbours up, no unexpected interface down, CPU and memory normal.
- Auto-rollback if post-checks fail.
What it must not do: push a wholesale startup-config replacement. That destroys local state - TACACS+ keys, licences, certificates - is not idempotent, and will eventually take a site offline. Converge declared state through idempotent modules so a re-run produces an empty diff.
Rollback: take a named checkpoint immediately before the change and restore it on post-check failure, then re-run the drift job to confirm convergence. Test the restore path and time it in the lab - do not assume five minutes until you have measured it, and measure it on the slowest model in the fleet.
2. How do you detect and handle drift, given that some drift is legitimate?
Read the worked answer
Detect: nightly read-only collection of running config versus rendered intent, into the same diff engine as the pipeline. Read-only polling means this job needs no write credentials and is safe at any frequency.
Classify before you act - this is the whole decision:
- Cosmetic (banner text, timestamps, session identifiers): ignore.
- Legitimate break-glass (an engineer changed a port during an incident): must be folded back into NetBox and the templates within an agreed window, or the drift job will keep flagging it and eventually revert something needed.
- Unauthorised (VLAN, ACL, routing change): ticket plus page.
Auto-remediate only a whitelist - NTP, syslog, SNMP, banners. Never auto-remediate routing, ACLs, or anything touching a spanning-tree or VSX state without a human: automated correction of a policy change can turn a localised mistake into a fabric-wide outage, and it destroys the evidence trail the auditors are asking for.
Feed it back upstream: if the same drift appears three times, the template or NetBox model is wrong, not the engineer. The drift report should generate design work, not just tickets.
3. How do you satisfy the audit requirements for credentials, approval and evidence without slowing the team down?
Read the worked answer
Credentials: vault-managed service accounts, rotating per site, injected only into the deploy job at runtime. CI holds no standing enable password and no write credentials for the drift job. Break-glass credentials stay under the existing sealed process and are logged when used.
Approval: deploy only from the main branch, only from a merged PR with an independent approver. The commit is the record of who approved and who executed; nothing else needs to be produced manually for the audit.
Evidence: TACACS+ command accounting forwarded to the SIEM captures every CLI command that ever touched a device, including break-glass. Pair that with the pipeline's own change log and checkpoint/rollback events, and the five-minute rollback claim becomes a measured metric reported per quarter, not a design assertion.
Do not slow the team down by making emergency changes go through the same lead time. Build a documented emergency path - break-glass, then retroactive PR within 24 hours - with the drift job reconciling it. Teams that lack an emergency path bypass the pipeline entirely, which is exactly how the original hand-edited VLAN incident happened.