Large Transfers Stall on a VXLAN Fabric With 9000-Byte Underlay
An eight-leaf, four-spine VXLAN fabric uses an EBGP underlay with four-way ECMP. Servers use 9000-byte MTU and are dual-homed to leaf pairs. After spine3 was replaced under RMA, large file transfers between servers on leaf2 and leaf7 stall at about 1.4 Gbps, while SSH and small pings pass. A ping with the DF bit set and an 8972-byte payload fails. All leaf-to-spine links are 100G and were configured with MTU 9216, except spine3's uplinks, which were left at the default 1500 after the replacement. The leaf IRB used for management is 1500. VXLAN encapsulation adds 50 bytes. The fabric is in production and you have a 45-minute window; spine3 currently carries about 25 percent of traffic.
1. Calculate the minimum underlay MTU for a 9000-byte server IP packet and explain why the 1500-byte spine link causes TCP to stall rather than fail outright.
Read the worked answer
Minimum underlay MTU is the 9000-byte server IP packet plus 50 bytes of VXLAN overhead, or 9050 bytes. The 9216-byte links are sufficient; spine3's 1500-byte uplinks are not. A 9000-byte TCP segment is emitted by the server with DF set, reaches spine3, and is dropped because it exceeds 1500. If ICMP 'fragmentation needed' is filtered, the sender never learns a smaller path MTU and retransmits the same large segment. Small packets and SSH pass, so the link looks healthy. The 8972-byte ping payload plus 28 bytes of IP/ICMP equals 9000 bytes, which becomes 9050 bytes after VXLAN and fails for the same reason. This is a black-hole, not a bandwidth limit; the 1.4 Gbps plateau is retransmission behavior.
2. Traffic between two servers sometimes pins to a single spine link even though four ECMP paths are up. What would you check, and why does VXLAN make this more likely?
Read the worked answer
VXLAN makes ECMP polarization more likely because all traffic between a given leaf pair uses the same outer source and destination VTEP addresses. If the hash uses only the outer 5-tuple, every flow between leaf2 and leaf7 hashes to the same spine link even with four-way ECMP. Check the hash configuration with show forwarding-options enhanced-hash-key and the ECMP/LAG hash policy, then inspect the forwarding table. The fix is to include inner IP/L4 fields in the hash for VXLAN-encapsulated traffic, or to use a hash policy that includes the inner header. Do not assume ECMP is broken just because show route lists four next hops; the forwarding hash determines utilization.
3. Give a safe fix sequence for spine3 within the 45-minute window.
Read the worked answer
First drain spine3 by shifting BGP preference or disabling its uplinks in a controlled step, so the 25 percent of traffic moves to healthy spines. Set mtu 9216 on spine3's leaf-facing and spine-facing interfaces, commit confirmed, and verify with show interfaces ... | match mtu. Test with a DF ping of 8972 bytes from leaf2 to leaf7 and an iperf run, then re-enable spine3 and confirm ECMP and large-flow throughput. If the MTU change bounces the interfaces, the drain prevents an outage. After the window, audit all RMA replacements for default MTU and add an MTU check to the acceptance checklist.