Zero-touch rollout of 180 branch VSP 4900 pairs with a non-idempotent template
An MSP is rolling out 180 branch sites, each with two VSP 4900 switches in a vIST pair. A Python/Ansible script renders a configuration from a site database and pushes it through ZTP. Each site gets 12 VLANs and 12 ISIDs from a pool starting at 40,000, with ISID = 40000 + site_id * 20. Nicknames are generated as 0.00.<site_id> for both switches in the pair. After 40 sites, 14 sites report duplicate nickname alarms. At 22 sites, vlan i-sid commands fail because the VLANs were not created. At 8 sites, the vIST pair forms but the second switch's ISIDs are not installed. The script is not idempotent and restarts from the beginning after any failure. The script has no pre-check or post-check phase and treats a CLI error as a warning.
1. Identify the design flaws behind the three failure classes.
Read the worked answer
The nickname formula is per site, but there are two switches per site. Both pair members receive 0.00.<site_id>, so every site is a duplicate-nickname site once both switches join the fabric. The fact that only 14 alarms appear after 40 sites reflects timing and which nodes had joined, not a formula that is safe.
The vlan i-sid failures are an ordering and idempotency problem. The template applies the ISID mapping before ensuring the VLAN exists, and a re-run starts from the beginning. If a previous run partially created a VLAN, the script may skip creation or use the wrong VLAN ID, and then the mapping fails. Treating the CLI error as a warning lets the site be marked complete with missing state.
The second-switch ISID failures are a per-node application problem. vIST does not automatically make every ISID and VLAN configuration identical on both switches. If the script targets only the primary or assumes the pair syncs all service configuration, the secondary has no local ISID mapping. Nickname uniqueness and ISID consistency must both be enforced per node, not per site.
2. Redesign the provisioning workflow to be idempotent and safe. Include pre-checks, ordering, and post-checks.
Read the worked answer
Make the site database node-aware: site_id, pair member A/B, management IP, loopback, system ID, and a nickname derived from site_id plus member index. Pre-check that the nickname and system ID are unused in the fabric, that the ISID block is free, and that the target VLAN IDs are either absent or match the intended configuration. Use create-or-verify logic for VLANs and idempotent commands for ISID mappings.
Order the push by dependency: management and IS-IS/SPBM base, B-VLANs, VLAN creation, VLAN-to-ISID mapping, vIST, SMLT, then FA. Apply every service command to both pair members. After each phase, query state and fail closed: show isis adjacency, show isis spbm i-sid, show vlan, show vist. Do not mark a site complete until both nodes pass. Add a rollback step that removes the new VLAN/ISID block and leaves SMLT down if vIST is not stable. Run the workflow first against a lab pair and one canary branch.
3. How would you remediate the 14 duplicate-nickname sites and the 8 sites with missing ISIDs without disrupting the whole fabric?
Read the worked answer
Fix the template and inventory first so remediation does not recreate the fault. For the 14 duplicate-nickname sites, identify the two claimants with show isis spbm and IS-IS logs, choose the intended nickname for each node, and change one node at a time in a maintenance window. Expect a transient IS-IS restart for that node; verify the fabric reconverges before moving to the next site.
For the 8 sites with missing ISIDs, compare show vlan and show isis spbm i-sid on both pair members, then apply the missing VLAN and ISID mapping to the node that lacks it. If the ISID exists but is not installed, check the B-VLAN and the VLAN binding before changing anything. Validate with a test host in the affected VLAN and with the post-check suite. Do not re-run the entire non-idempotent script against a live site; use a targeted remediation playbook with a before/after diff.