Large Packets Fail Across PODs in an EVPN-VXLAN Fabric
Your team runs a two-POD EVPN-VXLAN fabric built on QFX5120 leaves and QFX5120 spines. Each POD has two leaves, and every leaf connects to both spines over 100G links. Leaf-spine interfaces are configured with an IP MTU of 9000, access ports are left at the default 1500, and the underlay is OSPF. Servers in VLAN 100 (VNI 10100, subnet 10.10.100.0/24) use an anycast gateway on the leaves. A new application is split across POD A and POD B. Hosts inside the same POD communicate normally. A host in POD A can ping a host in POD B with the default 56-byte payload, but a 1400-byte ping with the do-not-fragment bit set fails. SSH between the two hosts connects, then stalls during key exchange. show evpn database lists both host MACs on both leaves with the expected remote flags, EVPN type-2 routes exist in both directions, and all OSPF adjacencies are full.
1. Which layer is the most likely fault, and which three checks would you run to confirm it before changing configuration?
Read the worked answer
The control plane looks healthy: MACs are learned remotely over EVPN and type-2 routes exist both ways, so the failure is almost certainly in the underlay forwarding path, specifically MTU. VXLAN encapsulation adds about 50 bytes to the original frame (outer Ethernet, outer IP, UDP, and VXLAN headers). A 1400-byte payload produces a frame near 1500 bytes, which becomes roughly 1550 bytes on the wire and exceeds any underlay link still at 1500. That explains why small pings pass, why DF-bit pings fail, and why SSH stalls once key exchange pushes larger packets.
Confirm it with three checks:
- Run
ping 10.10.100.20 size 1400 do-not-fragmentfrom the POD A host and from a leaf IRB; then step the size down to find the exact failure threshold. - Run
show interfaces extensive | match "MTU|drops|CRC"on every fabric interface, including LAG members. - Run
show configuration interfaces | display set | match mtuto find the link left at 1500, often a newly added spine uplink or a LAG member that missed the template.
2. Give the configuration change you would make and the order of operations. Why is raising the access-port MTU not the right primary fix?
Read the worked answer
Raise the MTU on the underlay only. On every spine-facing physical interface and on any aggregated Ethernet bundle in the fabric, set mtu 9000 on both leaves and spines. Leave the access ports at 1500 and leave the IRB at 1500, because the servers still send standard frames and do not need jumbo frames.
Order matters. Changing one end of a link first can break OSPF adjacency if MTU mismatch detection is enabled, so change both ends of a link back-to-back, one POD at a time, during a maintenance window. Use commit check, then commit confirmed 5 so an in-band mistake rolls back automatically, verify with show interfaces extensive | match MTU and a 1400-byte DF ping, then confirm with commit.
Raising the access MTU does not create underlay headroom: a server capped at 1500 still produces a frame that needs about 1550 bytes after encapsulation. It can also break hosts that cannot handle jumbo frames, which is why it is not the primary fix.
3. After the MTU fix, large pings succeed, but one application still stalls on bulk transfers between PODs while others are fine. What would you check next?
Read the worked answer
Hash-dependent failure points to inconsistent MTU or a drop on only some equal-cost paths. Check every ECMP next hop, not just the interfaces you already changed: run show configuration interfaces | display set | match mtu across both leaves and both spines, and show interfaces extensive | match "MTU|drops" on each spine-facing interface. A single spine link or LAG member left at 1500 will fail only the flows hashed onto it, which matches the symptom exactly.
Also look for policers or tcp-mss behavior on the IRB or uplink that could affect large TCP segments, and check interface drop counters for output drops rather than MTU errors. If all MTUs are consistent and counters are clean, move up the stack and check whether the application has an MSS or path-MTU discovery problem of its own, but rule out a single inconsistent underlay path first.