Broadcast storm on floor 3: why spanning tree did not save the access port

The loop is invisible to STP because nothing in it originates BPDUs. Spanning tree works by exchanging BPDUs and blocking the port that would create a loop. In this topology, the unmanaged switch is transparent: it has no STP implementation sending BPDUs toward the 424E, and the 424E's own BPDUs sent out port 12 arrive back on port 13 only after the loop is complete - by then, broadcast traffic is already circulating at line rate. So STP sees two access ports that each look like a normal single-device edge link.

Edge ports make it worse, and that is the important lesson. An edge port skips the listening/learning delay and goes straight to forwarding on link-up, because it is expected to face an end device. Turning on edge ports is correct for clients and phones - it avoids 30 seconds of DHCP delay - but it removes the delay that would otherwise give STP time to detect a BPDU and block the second port. Edge ports without BPDU guard (and without a loop-detection mechanism) convert a user mistake into an immediate, line-rate storm.

Why the whole floor, including phones, died The storm is flooded within VLAN 30, but the switch CPU spikes because every broadcast copy is punted for processing, and the FortiLink uplink saturates at 9.4 Gbps. Once the uplink is saturated, all VLANs sharing it - including the voice VLAN 40 - lose service. The symptom is not 'one VLAN is slow'; it is 'the floor is down', which is why the first responder often misdiagnoses it as a switch or uplink failure.

Diagnostic signature to remember: CPU 99% + uplink at wire rate + multiple ports at wire rate on a single switch + no STP topology change = a user-induced loop, not a hardware fault.

Treat every user-facing port as untrusted, and give the switch a way to react without a human.

1. BPDU guard on all edge ports. If a BPDU arrives on a port that should face an end device, the port is err-disabled rather than allowed to participate in topology. Combined with edge-port, this is the single highest-value control: a user plugging in a switch now isolates one port instead of a floor. Verify behaviour on a lab port before rolling out, and make sure the recovery path (auto-recovery timer or manual clear) is documented.

2. Storm control per port on broadcast, unknown-unicast and multicast, with a rate that reflects normal traffic on that port type (a desk port carries very little broadcast; a trunk carries more). Set the action to drop and trap, not just log - a storm that is merely logged still takes the floor down. Baseline normal rates for a week before choosing thresholds, or you will block legitimate traffic.

3. Port security / MAC limit on access ports (for example 2-4 MACs with sticky learning) so a user-added switch is limited in how much it can inject even before the storm-control threshold is reached.

4. Disable unused ports and park them in an isolated VLAN rather than leaving them in VLAN 30. This removes the easy wall-socket target and hurts nothing operationally.

5. Loop protection where supported (loop guard/loop-detect on edge ports), plus BPDU filtering only where you can predict the consequences - filtering BPDUs on edge ports is not a substitute for guard.

Standardise across all five floors: one port template (edge + BPDU guard + storm control + MAC limit + unused-port shutdown), one set of thresholds documented per port type, one alert path to FortiAnalyzer, and one recovery procedure with a named owner. Mixed per-floor settings are how the sixth floor becomes the next outage.

Also fix the cause, not just the symptom: the facilities team needed two ports. Give them two real patched ports or a small managed FortiSwitch - user workarounds are a design signal, not a discipline problem.

Minutes 0-3: contain before you diagnose Identify the switch whose ports are all at wire rate (here, floor 3) and stop the flood at its source, not at the FortiGate uplink - shutting the FortiLink uplink takes the whole floor out and removes your visibility. If you cannot find the loop instantly, shut the suspicious pair of access ports; the floor loses two desks instead of 180 users and the phones.

Find the loop point in the MAC table

• Look for the same MAC being learned repeatedly on two ports in the same VLAN, alternating between them. That is the loop signature: ports 12 and 13 both seeing the same source MAC.
• Cross-check per-port throughput and broadcast/unknown-unicast counters: two access ports at wire rate that should be quiet, and an uplink at wire rate.
• Confirm there is no matching STP topology change, which rules out a legitimate reconvergence.

Then confirm before you re-enable

• With one port still disabled, verify CPU drops to normal, the uplink returns to normal utilisation, and user traffic resumes.
• Check the switch event log for the port-up events at the start of the storm - the timestamp of two adjacent ports coming up within seconds is usually the moment the unmanaged switch was patched in.
• Physically locate the cable run for ports 12 and 13 and confirm they terminate in the same room. Do not re-enable until the second cable is removed or the port is reconfigured.

Evidence for the post-incident review

• Per-port counters and CPU graph for the 5 minutes before and after containment.
• MAC table snapshot showing the flapping MAC.
• Event log with port up/down timestamps and the identity of who patched what, if port descriptions or LLDP neighbours show it.
• The uplink utilisation and FortiLink traffic graph tying the floor outage to the storm.

Recovery rule: bring ports back one at a time with BPDU guard and storm control already applied, and re-check counters between each. Re-enabling both ports together re-arms the same loop before the new protections are proven.