VXLAN Cross-Rack Throughput Collapse on a New Leaf
A data centre runs two CE8850 spines and four CE6865 leaves. The underlay is OSPF area 0 on /31 point-to-point links; the overlay is iBGP AS 65010 with the spines as route reflectors, VXLAN with distributed anycast gateway. Servers in VLAN 10 use 10.10.10.0/24, mapped to VNI 5010, with L3 VNI 5000 for symmetric IRB.
A fifth leaf, leaf-5, was added for a new rack. Its two 25GE uplinks were configured with IP addressing, OSPF, BGP EVPN and the VLAN/VNI mappings. The fabric was originally built with mtu 1600 on every spine and leaf physical interface. The engineer who racked leaf-5 left the interfaces at factory defaults and added only Layer 3 configuration.
Servers on leaf-5 work well among themselves: iperf3 between two hosts in the same rack reaches 23 Gbps. Cross-rack traffic to leaf-2 stalls: SSH connects but freezes during large output, HTTP downloads settle near 1.5 Mbps, and iSCSI mounts to a target on leaf-2 time out. Ping from leaf-5 hosts to leaf-2 hosts succeeds at 0.4 ms; ping with a 1472-byte payload and DF set fails. No interface errors or drops are reported.
1. Explain the failure signature and quantify the encapsulation overhead that causes it.
Read the worked answer
The split between local and cross-rack behavior points to encapsulation. Traffic between two hosts on leaf-5 is bridged or routed inside the same leaf and never enters a VXLAN tunnel, so it runs at line rate. Traffic to leaf-2 is encapsulated by leaf-5's VTEP.
For an IPv4 underlay, VXLAN adds 50 bytes on the wire: 14 bytes of outer Ethernet, 20 bytes of outer IP, 8 bytes of UDP and 8 bytes of VXLAN. The outer IP packet must carry the inner 1500-byte frame plus 20 + 8 + 8 = 1536 bytes. With the outer Ethernet header, the frame on the link is 1550 bytes.
Leaf-5's interfaces are still at 1500, so a full-size encapsulated frame is oversized and is discarded. Small packets fit and are forwarded, which is why ping, SSH handshake and small HTTP responses work. Full-MSS TCP segments and iSCSI data PDUs are dropped, so transfers stall or time out. If the DF bit is set and the ICMP fragmentation-needed message never reaches the sender, TCP never reduces its MSS and the connection hangs rather than slowing down. Control-plane protocols stay healthy because OSPF and BGP packets are small, which is why no adjacency alarms were raised.
2. How would you prove the MTU problem with commands and tests before changing configuration?
Read the worked answer
Work from the interface outward:
- On leaf-5 and the attached spine ports, run
display interface 25GE1/0/1and read the MTU field. Expect 1500 on leaf-5 versus 1600 on the spine and on leaf-2. - From a leaf-5 host, run
ping -s 1472 -fto a leaf-2 host. It fails because 1472 + 28 = 1500 is the inner packet, which becomes 1550 after encapsulation. A 1400-byte payload should succeed, bracketing the failure threshold. - From leaf-5's VTEP loopback, run
ping -s 1550 -fto leaf-2's loopback. It should fail on the 1500 hop while succeeding once the path is 1600. - Confirm the overlay is otherwise healthy:
display bgp evpn peershould show all sessions Established,display vxlan tunnelshould show the tunnel to leaf-2 Up, anddisplay bgp evpn route-type mac-ipshould contain the leaf-2 host entries. This rules out control-plane causes before you touch MTU.
If some platforms count the oversized frames as input errors, note that counter; many count nothing at all, so absence of errors does not disprove the diagnosis.
3. State the fix and the verification, including any host-side consideration.
Read the worked answer
Set mtu 1600 on both leaf-5 uplinks and on the matching spine ports, and confirm every hop in the underlay path, including any transit or firewall link, supports at least 1600. Consistency matters: one 1500-byte hop is enough to reintroduce the drop. Then re-run the large-payload DF pings and confirm 1550-byte payloads pass.
For hosts that cannot perform PMTUD because ICMP is filtered, add MSS clamping on the VBDIF, for example tcp adjust-mss 1450, so TCP negotiates a segment size that fits the tunnel. This is a mitigation, not a substitute for an adequate underlay MTU; iSCSI and other non-TCP traffic still need real 1600-byte support.
Finally, add the MTU setting to the leaf provisioning template and to the acceptance checklist: display interface must show the fabric MTU on every uplink before a leaf is declared ready. Also verify that the fabric's peer links and any Layer 3 handoff to the WAN carry the same MTU, since the same encapsulation math applies beyond the leaf pair.