Change Control Fails Mid-Rollout on a Spine
You are rolling out a new BGP neighbor policy to 6 spines via a CloudVision change control. The change control has 6 tasks, one per device, executed sequentially. Task 3 (spine3) fails with a configlet error. Tasks 4-6 have not run. Spine3 is now partially configured: the new policy is present but the neighbor statements were not applied, leaving it in a state where it may not peer correctly. The change control shows Failed. You need to get the fleet consistent.
1. What are your immediate options for the failed change control, and which do you choose?
Read the worked answer
Options:
- Roll back the failed change control - CloudVision can revert the applied tasks to their pre-change state. This restores spine3 (and any already-applied tasks 1-2) to the prior config, giving a known-good baseline.
- Fix forward - correct the configlet error and run a new change control for the remaining devices plus spine3.
- Do nothing and manually patch spine3 - risky and breaks the audit trail.
Choose roll back first if the partial state on spine3 is unsafe (it likely is, since BGP neighbors are half-configured). Rolling back tasks 1-2 as well gives a consistent fleet, then you fix the root cause and re-run the whole change control. If tasks 1-2 are verified healthy and you are confident, you may roll back only spine3, but consistency across the fleet is usually worth the extra churn.
Either way, do not leave spine3 partially configured - that is the real hazard.
2. How do you find the root cause before re-running, and how do you prevent the same failure on the remaining devices?
Read the worked answer
Root cause: read the task's configlet diff and error output in the change control detail view. Common causes: a variable that doesn't resolve on spine3, a syntax difference, or a configlet ordering dependency (e.g., the policy must exist before the neighbor references it). Check spine3's device tags and firmware versus the other spines.
Prevention:
- Dry-run / preview the change control to see per-device diffs before executing.
- Check configlet compliance on all 6 spines first; fix any device that is out of sync so the change control starts from a clean baseline.
- If the failure is version- or tag-specific, use a configlet builder that branches per device, or split the configlet so the failing device gets the right variant.
- Re-run as a new change control (the failed one is an immutable record) and watch per-device status, not just overall task status.