Automation Rollout Causes Config Drift
An enterprise with 120 sites adopted template-driven orchestration for its SD-WAN fabric. Templates define interface, overlay, and QoS settings; site-specific variables cover circuits and LAN subnets. Over six months, engineers made 40 manual 'emergency' changes directly on devices during incidents. The orchestration platform still shows templates as the source of truth, but a compliance audit finds 18 devices whose running config differs from their template. Two of those differences silently disable a QoS shaper on high-utilization links. The team wants to reassert template control but cannot afford a mass outage from an unplanned push, and some manual changes may be legitimately needed.
1. Explain the risk of simply re-pushing templates to all 120 sites, and why the 18 drifted devices are the critical subset.
Read the worked answer
A blanket re-push would overwrite all manual changes, including any that were legitimate incident fixes, and could change interface/QoS behavior on links that are currently carrying production traffic. On high-utilization links, re-enabling a shaper that was manually disabled could throttle traffic and cause an outage; conversely, pushing a template that assumes a circuit that has changed could break overlay tunnels.
The 18 drifted devices are critical because:
- They are the ones where running state is not what the source of truth says, so any future automated push or audit will behave unexpectedly.
- Two of them have a silently disabled QoS shaper on high-utilization links - exactly the kind of difference that causes intermittent, hard-to-diagnose performance issues.
- They represent the gap between intent (templates) and reality, which is the core governance failure.
So the work is to reconcile drift deliberately, not to overwrite it blindly.
2. Design a safe reconciliation process that reasserts template control while preserving legitimate changes.
Read the worked answer
A staged, evidence-based process:
- Inventory drift: for each of the 18 devices, diff running config vs. template and classify each difference as (a) legitimate and needed, (b) stale/accidental, (c) unknown.
- Freeze and document: require change tickets for any further manual change; capture the 40 historical changes and map them to devices.
- Promote legitimate changes into templates/variables so the source of truth reflects reality. Do not re-push a template that contradicts a needed fix.
- Remediate in waves, lowest-risk first: start with non-production or low-utilization sites, then the 18 drifted devices, then the rest. Each wave: backup config, push, verify, roll back on failure.
- Handle the QoS shaper explicitly: confirm intended shaper values per link, set them in the template, and validate utilization/performance after push.
- Guardrails: enable config-drift detection/alerting and periodic compliance scans so drift is caught in days, not months.
3. What operational controls would you add so emergency manual changes do not silently accumulate again?
Read the worked answer
Controls that make manual change visible and temporary:
- Break-glass process: emergency changes allowed, but must be logged with a ticket and an expiry; the device is flagged for reconciliation.
- Drift detection and alerting: continuous comparison of running config to template, with alerts on any deviation and a dashboard of drifted devices.
- Change review: periodic review of all deviations; each is either promoted into the template or reverted.
- Policy-as-code / CI checks: template changes go through review and simulation before push.
- Audit cadence: monthly compliance report; quarterly full reconciliation.
- Ownership: name an owner for template governance so drift has an accountable resolver.
Together these convert silent drift into a tracked, time-bound exception.