Large Transfers Stall After a Top-of-Rack Switch Replacement
Proseware runs a 24-host NSX overlay. Each rack has its own /24 for TEP VMkernel interfaces, routed between racks with BGP, and the design specifies jumbo frames on all transport links. After a top-of-rack switch in rack B was replaced, users report that SSH and ping to VMs in other racks work normally, but SCP copies and database bulk loads stall or crawl and sometimes never finish. Traffic between VMs in the same rack is unaffected. Traceflow from a rack-A VM to a rack-B VM reports the injected packet as Delivered. vmkping between the TEP interfaces of two hosts in different racks succeeds at -s 1400 and fails at -d -s 1600. No NSX alarm fired.
1. Explain the failure pattern and name the fault. Why do small operations work while bulk transfers stall, and why did traceflow say Delivered?
Read the worked answer
The fault is an underlay MTU mismatch on the replacement switch's inter-switch links.
Why packet size is the dividing line. GENEVE encapsulation adds roughly 50 bytes on top of the guest frame (outer IP, UDP, GENEVE header, plus overhead accounting). A full-size 1500-byte guest frame becomes about 1550 bytes on the physical wire. If a link in the transport path is at 1500 MTU, that frame is dropped. Small packets - SSH keystrokes, ICMP echo at default size, session setup - fit comfortably and appear to work.
Why bulk transfers stall rather than fail cleanly. Large transfers depend on path MTU discovery or on fragmentation, and DF-bit blackholing is common in routed underlays. The sender keeps retransmitting full-size segments that never arrive, so the transfer crawls, then hangs. The connection itself is never refused, which is why it reads as a performance problem instead of a connectivity problem.
Why the rack boundary matters. Host-facing ports on the new switch took the jumbo configuration, so intra-rack encapsulated traffic fits. Only the inter-switch uplinks were left at the template default, so only cross-rack traffic is affected.
Why traceflow misled. Traceflow confirms that the forwarding path and policy verdict are correct for the packet it injects. Its default injected size is small, so it never exercises the MTU failure. Delivered is true and irrelevant.
2. What exactly must change, where must it change, and what is the operational cost?
Read the worked answer
Fix the underlay first.
- Restore jumbo MTU on the replacement switch's inter-switch and uplink ports to match the rest of the fabric - the design value, not the 1600 minimum. MTU must be consistent across every hop of the transport path; a single 1500-MTU link anywhere between two TEPs breaks large flows.
- Verify the physical path end to end, not just the two endpoints, including any routed transit links between racks.
Then verify the TEP side.
- The TEP VMkernel interface MTU must match the transport network. NSX requires the transport network MTU to be at least 1600 to carry 1500-byte guest frames; if any part of the fabric genuinely cannot do jumbo, 1600 is the floor, not 1500.
- Confirm the host switch uplinks and the VMkernel adapter both carry the correct MTU.
Operational cost. Changing the MTU on a TEP VMkernel interface is disruptive - the interface carries overlay traffic for every VM on that host. Plan it as a per-host change: maintenance mode with full evacuation, change, validate, exit maintenance, one host at a time per rack. Do not change TEP MTU and physical switch MTU in the same step, or you lose the ability to tell which change fixed or broke it.
3. How would you have caught this, and what monitoring gap does the "no alarm fired" detail expose?
Read the worked answer
The detection gap. NSX alarms and BFD are built on small control packets. BFD hellos, TEP heartbeat pings, and route advertisements all fit inside 1500 bytes, so every control-plane signal stayed green while the data plane was dropping full-size frames. Neither BFD nor a link-state alarm can see this fault. A quiet alarm panel is not evidence of a healthy underlay when the symptom set is size-dependent.
What would have caught it:
- A synthetic full-size probe between TEPs in every rack pair, run on a schedule and after any physical change: vmkping with the DF bit set and a payload sized so the total equals the transport MTU - for example
-d -s 1572for a 1600-byte frame once the 28 bytes of IP and ICMP headers are added. Small-payload pings prove nothing here. - An underlay MTU verification step written into the change procedure for switch replacement, firmware upgrade, and new link provisioning. Every port on the transport path should be compared against the design baseline before the change is closed.
- Per-flow throughput and TCP retransmit metrics from the workloads, not just link utilisation, so a stall pattern is visible before users report it.
Prevention is procedural: a replacement switch restored from a template is the single most common source of this fault. Treat the template as untrusted until its MTU, VLANs, and routing are compared with the design.