A 480-Router Push, a Broken Regex, and a CI/CD Proposal

Design backwards from "80 routers maximum":

1. Source of truth in Git, reviewed. Configuration generated from templates and intended-state data, not edited per device. Every change is a pull request with a second reviewer. This is what would have caught the regex: a reviewer sees lineinfile against a file with two ntp server variants and asks the question.
2. Render, don't patch. Replace lineinfile with a structured module (ios_config with explicit parents/lines) or a data model rendered to a full config. lineinfile is text substitution on a moving target; ios_config is idempotent and reports exactly what it would change. The specific bug - a pattern matching both ntp server and ntp server vrf Mgmt - is a class of bug, not an accident.
3. Dry run with diff, mandatory. --check --diff against the canary set, with the diff reviewed by a human. Output is attached to the change record.
4. Canary stage: 1 router in a lab, 2 routers in a non-critical region. Wait the full soak period (e.g. 30 minutes) and run post-checks.
5. Regional batches with hard failure limits. serial: 10, max_fail_percentage: 0 (or 5) so a single failure halts the run rather than continuing to 480. Region is a variable, so a run is scoped to one region and the blast radius is structurally capped at 80.
6. On-device safety net. Wrap the change in the platform's configuration rollback (configure terminal revert timer 10, with archive/configuration replace for the baseline). If the device is not confirmed reachable after the change, it self-reverts. This is what turns a nine-hour console recovery into a ten-minute timer.
7. Post-change verification that proves reachability, not just task success.

Why this catches the incident: the diff gate and the render-not-patch change would each have caught it independently, and the revert timer would have limited the damage to one batch even if both failed.

'ok' means Ansible sent the commands and the CLI accepted them. It does not mean the device is doing what you intended, or that it is still reachable in 30 minutes.

For the NTP change specifically, verification means:

• Reachability from the management network, tested from the orchestrator through the actual path (jump host, management VRF) - not from the device to itself.
• Intended state present: the NTP server entries are exactly the expected set for that device's role, with no duplicate or unintended ntp server vrf Mgmt line. Compare the rendered config against the running config, not against the playbook's return value.
• Functional state: show ntp status shows synchronized and a plausible stratum, and show ntp associations shows the expected peer. A device can hold a correct config and still be unsynchronised because the source is unreachable.
• No collateral damage: routes in the management VRF present, default gateway intact, OSPF/BGP neighbours up if applicable.
• Config diff snapshot archived per device, so the next change has a known-good baseline.

Design rules that follow:

• Verification is code with assertions, not a human reading output - otherwise it does not run at 03:00 on batch four.
• Verify against intent (data model), not against the previous config, or drift is carried forward silently.
• Every check must fail the pipeline and trigger rollback. A warning that nobody reads is not a gate.

Judgement: the difference between a script and an automation system is whether it can tell you that the change failed. Task return codes cannot.

Build the boring path first; resist the platform.

Build now:

1. Git with a fixed workflow - main protected, one branch per change, one reviewer, a PR template that asks "what is the rollback and what is the blast radius". This costs a day of training and prevents the largest class of incidents.
2. Ansible, not custom Python. Two of the four can support it; the other two can read a playbook in an afternoon. Custom Python raises the ramp cost and concentrates knowledge in two people - a bus-factor problem in a 480-device estate.
3. Serial batches, canary, revert timer, automated post-checks. These are configuration, not development, and they deliver most of the risk reduction.
4. Secrets handling that works through a jump host - a vault, credentials never in the repo, and an SSH path through the jump host tested as part of the pipeline rather than assumed.
5. A dry-run-only pipeline for the first month. --check in CI, human applies. Gets the team used to diffs before anything writes.

Deliberately not yet:

• A custom orchestration framework, an event-driven fabric, or full model-driven telemetry migration. Those need skills and time the team does not have, and they do not address the failure mode that actually occurred.
• Automating a process that is still changing. If the NTP/VLAN template is still being negotiated, freeze it first - automating churn just makes churn faster.

The real sequencing judgement: the incident was not caused by a lack of automation tooling. It was caused by an unreviewed, unverifiable text substitution applied without limits. A spreadsheet plus Git plus a revert timer would have prevented it. Tool sophistication is not the constraint; review and rollback are.