EVPN Anycast Gateway and Jumbo MTU on DCI
A data center uses a two-tier EVPN-VXLAN fabric with Nokia SR Linux leaves and spines. Two leaves, L1 and L2, share an anycast gateway 10.20.30.1/24 for VLAN 30. A dual-homed server uses an ESI-LAG to L1 and L2. The server's ARP cache shows the gateway MAC and works. A new DCI link connects the fabric to a 7750 SR PE over a 100 Gb/s transport; the PE uses an MPLS network with an MTU of 9000, while the SR Linux leaf uplinks are set to 9214. After the DCI link is commissioned, large flows between the server and a remote site fail, but small pings succeed. The server's MTU is 9000, and the remote CE is 1500. The fabric uses EVPN with ARP suppression. The operator sees VXLAN encap/decap counters increasing and some 'packet too big' messages.
1. Explain why small pings succeed and large flows fail, and where to check MTU.
Read the worked answer
Small pings at 1500 bytes fit everywhere. Large 9000-byte server frames get VXLAN encapsulation, adding about 50 bytes, so they need roughly 9050 bytes on the underlay. The leaf uplink is 9214, but the PE/MPLS path is 9000, so the encapsulated packet exceeds the transport MTU. If ICMP fragmentation-needed messages are filtered, PMTUD fails and TCP stalls. The 'packet too big' messages point to the PE or transport. Check MTU on the server, leaf port, VXLAN tunnel, PE network interface, MPLS, and CE.
2. What MTU values would you set on SR Linux and the 7750 SR, and what is the trade-off?
Read the worked answer
Either raise the transport/PE/MPLS MTU to at least 9050 plus label stack, or lower the server and VXLAN payload to fit 9000. On SR Linux, set the uplink and VXLAN tunnel MTU consistently; on the 7750 SR, set the network interface and MPLS MTU to match, and check SAP/SDP MTU. Keep the remote CE at 1500 and use TCP MSS clamping if needed. The trade-off is that raising transport MTU may affect all services and hardware queues, while lowering server MTU reduces throughput for large frames. Do not mix 9214 and 9000 on the same path.
3. Does ARP suppression explain the failure? What would you check instead?
Read the worked answer
No. ARP suppression only reduces ARP flooding; the server already has the anycast gateway MAC, and small pings work. The failure is size-dependent, so it is MTU/PMTUD, not ARP. Disabling ARP suppression would add broadcast traffic and would not fix the black hole. Keep it and fix MTU. Verify that the anycast gateway MAC is consistent on L1 and L2, but do not change ARP suppression for this symptom.