Brownfield Automation Across 4,200 Devices With a Unionized Change Process
Corvus Logistics operates 4,200 network devices: 2,100 Cisco IOS-XE switches, 1,400 Juniper MX and SRX platforms, and 700 Fortinet and Meraki devices. Twelve network engineers handle about 9,000 changes per year with a change success rate of 82% and an MTTR of 4.5 hours. Monthly audits find an average of 340 configuration drifts, 60% of them in access-layer VLAN and port settings. There is no IPAM source of truth; the CMDB is roughly 30% stale. Last quarter an outage was caused by a hand-typed ACL on a distribution switch. Leadership has funded two additional FTEs for automation and wants measurable improvement within 12 months, but the change-management process requires a human approval and a written rollback plan for every production change.
1. Lay out the first 12 months: what do you automate first, and what deliberately not?
Read the worked answer
Start where volume is high, blast radius is low, and rollback is trivial - not with the ACL that caused the outage. ACLs are the highest-consequence change type, and in month one there is no validation framework or rollback muscle to support them.
Phased plan:
- Months 1-3: build a read-only source of truth for one domain - access-layer VLAN and port configuration, which is 60% of the 340 monthly drifts. Discover existing state, define intended state, and produce a diff report with no push rights.
- Months 3-6: run drift detection in report-only mode and remediate a handful by hand using the generated diff. This builds trust and calibrates the templates against reality.
- Months 6-9: automate the ten highest-volume change types - port VLAN moves, description and naming standards, NTP and syslog, standard ACL objects - behind a human approval gate with automated pre- and post-checks.
- Months 9-12: extend to firewall and ACL objects with validation, and add auto-rollback on failed post-check.
What not to do: build a multi-vendor abstraction layer before one vendor pipeline works. Standardize intent for the 2,100 IOS-XE devices first; they are the largest population and the fastest source of measurable wins. Also do not let the CMDB become the source of truth - at 30% stale it would poison every template.
2. What should be the source of truth and transport, given three vendors and a stale CMDB?
Read the worked answer
Separate the data model from the transport, because they have different constraints.
Source of truth: a structured store - NetBox or similar for IPAM and device inventory, plus Git for intended configuration and templates. Build it from discovery for the current state, then require every change to pass through it. Importing the CMDB is not an option at 30% stale; treat the CMDB as a downstream consumer.
Transport, per platform, chosen on model coverage rather than ideology:
- IOS-XE: NETCONF/YANG for structured resources where the model exists; CLI as a fallback for gaps, but every CLI template is a liability because it has no schema validation.
- Juniper: NETCONF or gNMI with OpenConfig or native YANG; the model coverage is good enough that CLI should be rare.
- Fortinet and Meraki: vendor APIs. Meraki is dashboard-API only and will never be fully declarative - accept that and scope it to the operations it supports.
Design rule: one renderer per platform, one shared intent schema above them. Do not build a lowest-common-denominator abstraction - it will not survive contact with vendor-specific features and will be abandoned within a year.
Validate every rendered configuration against a schema and a policy check before it reaches a device; catching a wrong VLAN in CI is free, catching it in production is the outage you are trying to prevent.
3. How will you measure success, and how do you fit the human approval and rollback requirement into an automated pipeline?
Read the worked answer
Metrics, baselined now and reported monthly: change success rate (82% to 95%), MTTR (4.5 hours downward), drift count (340 to under 100), percentage of changes executed through the pipeline (target 30% in 12 months), and time to provision a standard access port. Avoid vanity metrics such as lines of code or number of playbooks.
Fitting the approval gate: use a plan-and-apply model. The pipeline generates the change record, the exact diff, the pre-check results, and a machine-generated rollback plan; a human approves the plan, not the keystrokes. This satisfies the written-rollback requirement better than today's process, because the rollback is generated from the observed before-state rather than typed from memory.
Safety mechanics: every apply runs a pre-check (device state matches the assumed before-state, otherwise abort), a post-check (intended state achieved, protocols up, no new alarms), and automatic rollback on failed post-check. Prove it in a lab and on five devices for 60 days before any production scope.
Sequencing risk: the two new FTEs should own the pipeline and the intent model, not the day-to-day changes. If they become the queue for every change request, the program stalls at the first busy month.