VXLAN Fabric - Wrong Suspect, Right Problem (MTU)
A small data center runs VXLAN over an OSPF-routed underlay; every underlay link has MTU 1500. Leaf1 (VTEP 10.0.0.1) and Leaf2 (VTEP 10.0.0.2) use static VTEP peers with no multicast. VNI 5010 maps to VLAN 10 on Leaf1 and to VLAN 20 on Leaf2; both VLANs carry the same subnet 172.16.10.0/24 with anycast gateway 172.16.10.1. VMs on Leaf1 ping their gateway fine and can ping VMs on Leaf2 with small packets (ping -s 1000 succeeds), but SSH to a remote VM hangs right after the banner and large file copies stall at a few hundred kilobytes. A junior engineer proposes remapping Leaf2 to VLAN 10 and lowering the underlay OSPF cost.
1. Evaluate the junior engineer's proposed change. Is the VLAN 10 / VLAN 20 difference the fault? Justify your answer.
Read the worked answer
No - and applying the change would be a mistake for two reasons.
The VLAN difference is not the fault. In VXLAN the local VLAN ID is locally significant; it is only a local attachment identifier between an access port and a bridge domain. The end-to-end identifier is the VNI. VLAN 10 on Leaf1 and VLAN 20 on Leaf2 both mapping to VNI 5010 is the documented, supported design - it is precisely how you keep the same tenant separated from other tenants on each leaf's local trunking. Remapping Leaf2 to VLAN 10 would be harmless at best and could collide with another tenant already using VLAN 10 on Leaf2.
The evidence points at MTU, not segmentation. Both leaves are in VNI 5010 and both carry 172.16.10.0/24, so the fabric is delivering packets - small ones. The pattern is the classic MTU signature: ping -s 1000 (payload 1000 + 8 ICMP + 20 IP + 14 Ethernet = under the limit) succeeds; TCP handshakes and SSH banners are small and succeed; the first full-size data segment is dropped, so SSH hangs after the banner and file copies stop at the point where the window opens up.
Lowering OSPF cost would change nothing except path selection - there is only one path anyway.
State the arithmetic: outer IP 20 + UDP 8 + VXLAN header 8 + inner Ethernet 14 = 50 bytes of overhead, so a 1500-byte underlay yields an inner MTU of 1450.
2. Give the remediation for the MTU problem, including the options you would reject and why, and the verification.
Read the worked answer
Root cause: inner frames up to 1500 bytes are encapsulated into 1550-byte outer frames that the underlay cannot forward.
Preferred fix - raise the underlay. Set MTU 1600 (jumbo) on every link the encapsulated traffic crosses - leaf uplinks, spine-to-spine, and any transit device - then set the VTEP/NVE interface MTU and the VMs' MTU accordingly. 1600 gives 1550 plus headroom for the anycast/tenant options and future overlays.
Fix if jumbo is impossible end to end: reduce the inner MTU, i.e. set the VM NIC / tenant gateway MTU to 1450 and clamp TCP MSS to 1410 or below on the tenant interfaces. This works but permanently costs 50 bytes of payload on every tenant flow and must be applied consistently to every host and every gateway, or you get a partial fix where some flows stall and others do not.
Rejected options: remapping VLANs (does not address MTU); lowering OSPF cost (no alternate path); fragmenting the outer packet (VXLAN expects the underlay to carry full frames; fragmentation is widely discouraged and often dropped by the physical fabric).
Verification: ping from VM to VM with DF set and escalating sizes (ping -s 1472 should fail, -s 1400 should succeed) before and after; check display interface on every underlay hop for the configured MTU; display vxlan vni 5010 and display bridge-domain to confirm the VNI stays up across the change; and re-run an SSH session plus a large file copy, because those fail in the window where ping still passes.
3. The team wants an automated pre-change check, run from Python (NETCONF/Netmiko), to catch this class of fault before a fabric change. What should the check collect, and why is a ping from the VTEP insufficient?
Read the worked answer
What to collect, per device: hostname and management IP; the MTU of every interface in the underlay path (display interface / the NETCONF ietf-interfaces mtu leaf), including the NVE/tunnel interface; the VNI-to-VLAN-to-bridge-domain mapping; and the VTEP peer list. Then assert two invariants: every underlay interface along a VTEP-to-VTEP path carries a uniform MTU ≥ VTEP MTU + 50, and every VNI referenced by a bridge domain has a resolved peer on the remote leaf.
Why VTEP ping is not enough: an ICMP echo from the VTEP is generated by the control plane and is small. It tests reachability, not forwarding capacity for encapsulated tenant frames. It will pass on a fabric that cannot carry a single full-size tenant packet. Worse, an echo sourced from the VTEP never gets encapsulated at all, so it does not even exercise the overlay path.
Better probes: a size-and-DF probe across the underlay (ping -s 1550 -f) between VTEP loopbacks, plus a per-hop MTU collection so you get the failing device's name, not just a failure. Read-only first: gather and report, do not remediate. Use the candidate datastore with confirmed-commit for the actual change, so a misconfiguration that breaks management access rolls back automatically; and log every pre-change value so the rollback is a restore of known state rather than a guess.