EVPN Multi-Site Jumbo Frame Drops After Border Leaf Addition
An enterprise operates two data centers: DC-A with 24 Nexus 9336C-FX2 leaf switches in 12 vPC pairs and DC-B with 12 leaf switches in 6 pairs. Both sites use OSPF underlay and eBGP EVPN overlay; each site has two border gateways. After a new border-leaf pair was added to DC-A and a new VLAN was stretched to DC-B, approximately 2% of TCP flows between racks in DC-A intermittently drop, and ping with 2000-byte payload fails between hosts in the same VLAN on different leaf pairs. ARP suppression is enabled on all SVIs. The spine uplinks are configured with system mtu 9216; the new leaf pair uses default 1500 MTU. One storage host is attached to an orphan port because its vPC peer link failed earlier and was not restored. The team wants to add a second border-leaf pair in DC-B and enable anycast gateway.
1. Which design and configuration issues best explain the 2000-byte ping failures, and what verification steps would you take before changing any border-gateway design?
Read the worked answer
Start with the forwarding path, not the border gateways. A 2000-byte payload needs at least 2050 bytes of IP-level MTU after VXLAN encapsulation, because VXLAN adds roughly 50 bytes of outer IP, UDP, and VXLAN headers. If the new leaf pair is at the default 1500 MTU, the frame is dropped or fragmented at the access leaf before it reaches the 9216-MTU spine. Verify with show interface ethernet 1/1, show system mtu, show running-config interface, and a ping with the DF bit set and a 2000-byte payload from both hosts. Check the SVI ip mtu and the VTEP MTU as well; a 9216 underlay does not help if the SVI or access port is 1500. ARP suppression can make the ARP resolution look healthy while the data plane still fails, so do not treat it as the root cause. Fix by setting system mtu 9216 on the leaf pair, aligning the SVI MTU with the host jumbo-frame setting, and retesting. The orphan port is a separate availability risk and should be repaired, but it does not explain the same-VLAN jumbo-frame failure.
2. Recommend an anycast-gateway and border-gateway placement design for seamless subnet mobility between DC-A and DC-B, and state one risk.
Read the worked answer
Put the anycast gateway on every leaf that hosts the stretched subnet, using the same virtual IP and virtual MAC on all leafs. That way a VM keeps its default gateway after vMotion and does not depend on a centralized first-hop router. The border gateways should not be the first hop for local VLANs; they should provide inter-site VXLAN and external connectivity only. In DC-B, add the second border-leaf pair for redundancy, but keep both pairs in the same multi-site domain with consistent route targets, VLAN-to-VNI mappings, and site IDs. Validate that the anycast MAC is identical on both sites for the same subnet; a mismatch can blackhole ARP or ND replies after a live migration. One risk is BUM replication volume: stretching a subnet across sites means broadcast, unknown unicast, and multicast traffic may traverse the border gateways, so scope the stretched VLANs tightly and monitor border-gateway replication counters.