Duplicated BUM traffic after a PE reboot in an EVPN all-active multihomed service
An operator aggregates 240 business access switches onto a pair of 7750 SR-7 PEs using EVPN-MPLS all-active multihoming. Every access switch is dual-homed with one LACP bundle (ESI 00:11:22:33:44:55:00:01) to PE-A and PE-B and belongs to a VPLS service with EVPN. PE-A is rebooted for a software upgrade. After it returns, known unicast forwarding is correct and ECMP across both PEs looks healthy, but the NMS reports duplicated broadcast and multicast delivery to 38 access switches, plus continuous MAC moves for 12 hosts between the two PEs. The ESI, LACP system ID and LACP keys match on both PEs, and ES preference is default on both. During the upgrade the access bundle on PE-A was left administratively up. Two route reflectors serve the EVPN address family, two hops away.
1. Explain how duplicated BUM delivery can coexist with correct known-unicast forwarding, and separate the checks that distinguish a designated-forwarder fault from an ES-route acceptance fault.
Read the worked answer
Known unicast and BUM take different paths through the logic.
Known unicast: each PE learns the source MAC locally on its own access bundle and advertises an EVPN MAC route. Each PE forwards a frame for a locally learned MAC straight out of its local attachment circuit, so unicast works even if the two PEs disagree about whether they share an Ethernet segment.
BUM: within an Ethernet segment, only the designated forwarder for that segment and that service instance floods into the access bundle; the non-DF drops. If both PEs believe they are DF, both flood, the NMS sees duplicates, and because each PE learns and re-advertises the same host MAC from a different next hop, the MAC routes flap and the 12 hosts show continuous moves. That is exactly the pattern when one PE does not consider the segment shared.
Distinguishing the two faults:
- DF fault: both PEs see the ES route and the ESI is exchanged, but the election is wrong, for example because service carving or ES preference differs, or the DF state was recomputed with a stale ordinal after the reboot.
- ES-route acceptance fault, the more likely one here: the ES route is generated but rejected on one PE because the EVI import route-target set differs after the reboot restored an older configuration, or the next hop is unresolvable. Then each PE treats the segment as single-homed and computes itself DF.
Check show service id <id> evpn-mh DF state per service, the received ES routes in the EVPN RIB, the import and export route targets on the EVI, and next-hop resolution toward the two route reflectors before touching election parameters.
2. What does the MAC-move pattern tell you about forwarding state, and what would you do differently during the next PE-A upgrade?
Read the worked answer
The MAC moves are a symptom of the same flooding fault, not a separate loop. When both PEs flood BUM into the segment, a broadcast or unknown-unicast frame from a host behind the access switch is seen by both PEs, so both learn it locally and both advertise it. The remote PE receives its own peer's MAC route for a MAC it just learned locally, which triggers a move; the next frame reverses it. The move count therefore grows with broadcast rate, which explains why it concentrates on the 12 chatty hosts.
Implications: do not treat this as a loop and reach for STP or MAC limiting first, because neither will stop it. Confirm the diagnosis by checking whether the move counter increases in step with broadcast counters on the access bundle.
For the next upgrade:
- Administratively shut the access bundle, or the ESI, on PE-A before the reboot so that PE-B is unambiguously the only forwarder, then restore it after PE-A has synchronized and re-advertised its ES and MAC routes.
- Lower the ES preference on the PE being upgraded if a faster, scriptable method is preferred over shutting the bundle.
- Add a post-change validation step that reads DF state, ES-route presence and MAC-route counts before declaring the maintenance complete.
- Compare the running configuration battery of the EVI against the golden configuration immediately after the reboot; a route-target mismatch introduced by a restored older configuration is a process failure, not a protocol failure.
3. The automation team wants an automated post-upgrade check for this service. Which checks would you require, and which would you deliberately leave out?
Read the worked answer
The check should validate the state that actually failed, not just that interfaces are up.
Require:
- DF state per EVI per ES on both PEs, and an assertion that exactly one PE is DF per service - this catches the fault directly.
- Presence and acceptance of the ES route in the EVPN RIB on both PEs, including next-hop resolution to the route reflectors; an ES route that is generated but not imported is the failure mode that started this incident.
- MAC-route count per EVI compared against a learned baseline, plus a delta on the MAC-move counter over a short sampling window. A move rate above a small threshold should fail the check.
- BUM counters on the access bundle on both PEs, to catch duplicate delivery in the data plane rather than inferring it from the control plane.
Leave out:
- Generic interface up/down and reachability pings, which would have passed throughout this incident and give false confidence.
- Any check that writes configuration to resolve a mismatch. The check should report and block, not remediate, because a wrong automatic fix on a segment serving 240 access switches has a much larger blast radius than a delayed maintenance closure.
- Deep per-host MAC verification, which is brittle and produces noise rather than signal.