CCIE Enterprise: HSRP, STP and the Maintenance Window That Wasn't

HSRP, STP and the Maintenance Window That Wasn't

Two Catalyst 9500 distribution switches, D1 and D2, serve four access stacks in a campus of about 3,000 users. HSRPv2 runs per VLAN: D1 is active for VLANs 10 and 20, D2 for VLANs 30 and 40, with preempt enabled on both. STP root follows the same split (D1 priority 4096 for 10/20, D2 priority 4096 for 30/40). Uplinks to the core are routed /30s in OSPF area 0. During a maintenance window the D1 uplink to the core was shut. VLAN 10 users lost connectivity for roughly 35 seconds before recovering via D2. One week later a newly racked access switch was trunked in, and the campus saw about 30 seconds of degraded forwarding; its STP priority was left at default and no storm-control alarms fired.

1. The 802.1w/HSRP timers should not produce a 35-second outage on their own. Explain what actually happened for the first ~25 seconds of it, and give the configuration you would add to D1, with values.

Read the worked answer

The uplink going down changes nothing in HSRP. D1 keeps priority 100 (or whatever is configured), stays Active, and keeps answering ARP for the virtual MAC - so VLAN 10 hosts keep sending frames to D1. But D1's OSPF adjacency to the core died, and the static/default route it was relying on is gone, so D1 drops the packets in hardware. That is a black hole, not a delay: traffic is being delivered to a switch that has nowhere to send it.

What you would see to confirm it: show standby brief shows D1 still Active after the link is shut, and show ip ospf neighbor shows the uplink adjacency gone. If instead HSRP had failed over immediately, the loss would have been ~10 s (hold time) plus ARP/MAC convergence. So the 35 s is ~25 s black hole + ~10 s of legitimate HSRP/ARP convergence once D2 took over.

The fix is to make HSRP state follow the uplink state with a deterministic margin:

interface TwentyFiveGigE1/0/1
 description UPLINK-TO-CORE
!
track 10 interface TwentyFiveGigE1/0/1 line-protocol
 delay down 5 up 30
!
interface Vlan10
 standby version 2
 standby 10 priority 110
 standby 10 preempt delay minimum 120
 standby 10 track 10 decrement 30

Reasoning on the numbers: D2 must be at the default 100, so D1 at 110 with a decrement of 30 lands at 80 - decisively below D2, not a one-point margin that simultaneous tracking events could tie. preempt delay minimum stops D1 grabbing the active role back before OSPF and the FIB have settled, which otherwise creates a second, shorter outage on the way back.

2. The newly racked switch caused a campus-wide forwarding disruption despite being a single access device. Give the most likely mechanism and the two lines of defence you would apply.

Read the worked answer

Bridge ID is priority plus MAC, and on any VLAN where both D1 and D2 were left at the default 32768, the election is decided purely by MAC address. A newly racked switch with a low MAC therefore wins root for those VLANs - typically the voice VLAN, an out-of-band management VLAN, or a newly created segment that someone forgot to add to the primary/secondary root pair. When a new root appears, every bridge in that VLAN flushes its MAC table and re-runs listening/learning, which is exactly the 30-second hole reported, and flooding during that window is what users perceive as an outage.

The tell is the VLAN scope: VLANs 10-40 were fine because they have explicit 4096 priorities; the affected VLANs were the ones nobody had pinned.

Defences:

Make root placement explicit and exhaustive - spanning-tree vlan 10-40,50,99 root primary on D1 and root secondary on D2 (which yields 4096/8192 style priorities), and audit that every VLAN in use is covered, not just the ones in the original build sheet.
Never let an edge device participate in root election - spanning-tree guard root on the distribution ports facing access switches, plus spanning-tree portfast bpduguard default on true edge ports. Root guard puts the offending port in root-inconsistent state instead of reshaping the topology.

Verification is per-VLAN, not global: show spanning-tree root and show spanning-tree vlan 99 must show the expected bridge as root with the expected priority, on every VLAN.

3. You now rely on HSRP tracking. A later incident involves a core link that stayed up but degraded to 40 percent loss for ten minutes, and HSRP never moved. Was tracking the wrong tool, and what would you add?

Read the worked answer

Interface line-protocol tracking is binary: up or down. A lossy but up link reports up, the tracked object stays up, and no decrement happens - so nothing in the HSRP design reacts. That is not a bug in the config, it is the wrong signal for the failure mode.

For a degraded path you need an object that measures reachability rather than carrier state:

ip sla 20
 icmp-echo 10.0.0.1 source-interface TwentyFiveGigE1/0/1
 frequency 5000
 threshold 500
 timeout 1000
!
ip sla 20 schedule life forever start-time now
!
track 20 ip sla 20 reachability
 delay down 15 up 60
!
interface Vlan10
 standby 10 track 20 decrement 30

Two judgement points. First, the down delay must be longer than the failure you care about - a 15 s delay absorbs a couple of lost probes instead of flapping the default gateway on a single dropped ICMP. Second, the probe destination must be something that answers ICMP and reflects the path you are protecting; probing a directly connected neighbour tells you almost nothing. Keep the line-protocol tracker as well, so a hard failure still converges immediately, but accept that partial degradation will only be caught at the probe interval.