MSTP region mismatch after an aggregation switch replacement
Four Nokia 7250 IXR-R6 switches form a metro aggregation ring SW1–SW2–SW3–SW4–SW1 on 10G links, carrying 40 VLANs for residential and business services. MSTP runs with region name METRO-AGG, revision 3, instance 1 for VLANs 10-19, instance 2 for VLANs 20-39, and the CIST for the rest. SW3 fails on a Friday night and is replaced with a cold spare from the warehouse. The technician copies the VLANs and instance mappings from the template but types the region name METRO_AGG and revision 1. On Monday, NSP reports the SW2–SW3 link at 92% average utilization, and the helpdesk has intermittent loss reports from VLAN 25 customers behind SW4. show spanning-tree mst config on SW3 reports a configuration digest different from SW1, and show spanning-tree mst 2 shows both of SW3's ring ports forwarding while the same instance on SW1 blocks a different port. No loop protection or BPDU guard is enabled on the ring. The next change window is Tuesday 01:00-04:00.
1. What exactly does the region mismatch do to the forwarding topology, and why does VLAN 25 suffer while VLAN 12 looks healthy?
Read the worked answer
MSTP defines a region by name, revision and VLAN-to-instance mapping; the digest must match for a switch to belong. With METRO_AGG/rev 1, SW3 is a one-member region of its own.
- Across a region boundary only the CIST is shared. The per-instance trees (instances 1 and 2) are computed separately inside each region, so the designed per-VLAN blocking points no longer line up around the ring.
- No immediate storm, but a broken tree. The CIST still blocks one port somewhere, so the ring does not flood, but instance 2 can forward on both of SW3's ring ports. VLAN 25 traffic between SW4 and SW2 is hairpinned around the ring instead of using the designed blocked port - hence 92% on SW2–SW3 and loss only for instance-2 customers.
- Latent loop. Instance 2 now has no blocking port on SW3 at all; if the CIST blocking port elsewhere fails, instance 2 loops and floods.
- Why VLAN 12 looks fine: it is in instance 1, whose intended blocking point happens to coincide with the CIST blocking point, so it is a misleading health signal.
2. Give the remediation sequence for the Tuesday window, including what you verify before touching SW3 and how you prove the fix.
Read the worked answer
- Before the window: capture
show spanning-tree mst config,show spanning-tree mst <n>and interface counters on all four switches; confirm the correct digest and the intended per-instance blocking ports from the design document. Save a copy of SW3's configuration. - Do not change SW1, SW2 or SW4 to match the wrong region. SW3 is the odd one out; aligning the other three to it would spread the defect.
- Isolate before correcting. Shut SW3's two ring ports (or disable MSTP on them) so the region correction cannot create a transient loop while digests reconverge.
- Correct and pre-verify. Set region name METRO-AGG and revision 3 on SW3; confirm the digest now matches SW1 before re-enabling the ports.
- Bring the ring back one port at a time, watching topology-change counters and the per-instance forwarding/blocking state.
- Prove the fix: all four digests identical, instance 2 blocking port on the designed switch and port, SW2–SW3 utilization back to baseline, and end-to-end loss tests from VLAN 25 and VLAN 12. Rollback is restoring the saved SW3 config if any step misbehaves.
3. What design and operational controls would you put in place to prevent this class of error on future replacements?
Read the worked answer
Governance first, then architecture:
- Generate, don't type. Build the MSTP block (region name, revision, instance map) from a source of truth and deploy it with NSP/NFM-P templates, so a spare inherits the exact values.
- Make the digest a compliance check. Pre- and post-change
show spanning-tree mst configcompared against a golden value, alarmed in NSP on mismatch - discovery by helpdesk is too late. - Contain the blast radius. Enable loop protection, BPDU guard on edge ports and root guard on the ring, and prune unused VLANs from trunks; a future mismatch should be contained, not just detected.
- Pre-stage spares. Keep a warehouse spare pre-loaded with the ring baseline and a documented burn-in checklist that includes the MSTP digest.
- Reduce dependence on STP. Longer term, an EVPN-MPLS ring with ESI multihoming, or a fully routed ring, removes the per-VLAN spanning-tree failure mode altogether.