EVPN Fabric Under ARP and BUM Pressure
A service provider runs a VXLAN EVPN fabric with four leaf pairs and two spines to support 1,500 VLANs and about 6,000 virtual machines. The underlay is OSPF with point-to-point links; the overlay uses ingress replication. Each leaf pair has a vPC domain, and anycast gateway is enabled. During a recent expansion, the NOC sees intermittent MAC moves on leaf-3, ARP cache utilization at 91% on two leaves, and BUM traffic at 3.2% of total. Applications report 200–400 ms stalls every few minutes. The team has added 200 VLANs in a month and plans 500 more. A vendor recommends disabling anycast gateway and using multicast in the underlay. The change board wants a diagnosis and a design decision before the next maintenance window. The fabric has no ARP suppression enabled on several VLANs because of an earlier interoperability issue.
1. Diagnose the likely cause of the MAC moves and application stalls, and specify what evidence you would collect.
Read the worked answer
The combination points to a control-plane/data-plane scaling problem rather than a single bad link. MAC moves on one leaf usually mean the same MAC is being learned from two VTEPs or from a vPC peer link and an orphan port. Check the EVPN MAC mobility sequence and whether the moves are between the vPC pair (expected during failover) or between unrelated leaves (duplicate IP, misconfigured LACP, or a VM NIC team). The 91% ARP cache is more urgent: with 1,500 VLANs and 6,000 VMs, ARP suppression may be off or ineffective, so every unknown IP triggers flooding. BUM at 3.2% with ingress replication means the originating VTEP is replicating to every remote VTEP and burning CPU; the 200–400 ms stalls are consistent with ARP/unknown-unicast floods filling queues. Collect per-VTEP replication counters, leaf CPU, EVPN route counts, MAC mobility events, and a short ERSPAN capture of ARP. Also verify MTU end-to-end and any orphan-port configuration. Do not start by disabling anycast gateway; that would increase ARP flooding.
2. Decide whether to keep anycast gateway and whether to move from ingress replication to multicast, explaining the tradeoffs.
Read the worked answer
Keep anycast gateway. It is not the cause of the MAC moves; disabling it would force traffic through a single gateway VTEP and increase latency and ARP load. The real decision is BUM replication. Ingress replication is simple and works at this scale, but 3.2% BUM with 1,500 VLANs and growing is close to the point where head-end replication CPU becomes the bottleneck. Multicast in the underlay can offload replication, but it adds PIM, RP placement, and group planning, and it is not a fix for ARP suppression. My recommendation: enable and verify ARP suppression first, segment the largest VLANs into smaller subnets, and set a threshold: if BUM remains above 3% after suppression or if any VTEP sustains more than 70% CPU, move to multicast for the high-BUM VLANs. Do not rip and replace the underlay for a 3.2% symptom; fix the flooding source.
3. Propose scaling guardrails before the team adds 500 more VLANs.
Read the worked answer
Before adding 500 VLANs, establish guardrails. Per-leaf limits: MAC table, ARP/ND entries, EVPN routes, and VLAN-aware bundles; 1,500 VLANs on four leaf pairs already means roughly 375 VLANs per pair, so check the platform's scale numbers, not just the datasheet maximum. Set a BUM budget per VLAN and per VTEP, enable ARP suppression everywhere, and require a /24 or smaller per subnet to cap ARP. Use tenant VRFs to contain failures and avoid a single VLAN numbering domain. For the new 500, require an application owner, a subnet plan, and a BUM estimate before provisioning. Track MAC mobility as an alert, not a log. If the fabric cannot meet the per-leaf EVPN route scale with headroom, add a leaf pair before adding VLANs; scale-out is cheaper than a 10 a.m. outage.