vSAN MTU Mismatch After a Switch Firmware Upgrade
A six-node VxRail cluster uses dual 25GbE ToR switches and a vDS. The vDS uplinks and physical switch ports were set to MTU 9000 during a recent switch firmware upgrade. The network team also set the vMotion VMkernel to 9000 but left the vSAN VMkernel at 1500. vSAN health now reports 'MTU mismatch' and intermittent 'vSAN cluster partition' alarms. VM provisioning is slow, and vSAN resync traffic is saturating the 25GbE links. The customer wants to change the vSAN VMkernel MTU to 9000 on all six hosts immediately during business hours. No host is in maintenance mode, the cluster uses vSAN FTT=1 RAID-1, and capacity is at 70%. A change freeze starts in 48 hours, and the customer argues that an MTU change is non-disruptive because it is only a VMkernel setting.
1. How would you confirm that the vSAN VMkernel MTU is the cause, and what evidence would you collect?
Read the worked answer
Use the vSAN VMkernel as the source and test the largest frame with the DF bit set: vmkping -d -s 8972 <peer vSAN vmk> because 8972 payload plus 28 bytes of ICMP/IP equals 9000. If large pings fail but 1472-byte pings pass, the path MTU is inconsistent. Repeat from each host to every other host's vSAN VMkernel, not just one pair, and compare with vMotion. Collect the VMkernel MTU, the vDS and portgroup MTU, uplink MTU, and the physical switch port MTU. Check vSAN health for the exact MTU mismatch alarm and check NIC error and drop counters. Intermittent partitions can also come from uplink hashing or a single bad optic, so correlate by host and uplink; do not assume MTU from the alarm alone.
2. What remediation order minimizes risk, and should this be done during business hours?
Read the worked answer
Do not do it during business hours on a 70%-full FTT=1 cluster. Although MTU is 'only a VMkernel setting,' changing it can briefly disrupt vSAN traffic and trigger APD or partition alarms; at 70% full, a resync storm is a real risk. Schedule a window, verify all six hosts, both ToR switches, the vDS, portgroups and uplinks are consistently 9000, and take a configuration backup.
Set the vSAN VMkernel MTU to 9000 one host at a time, then immediately vmkping from that host to all peers and watch vSAN health before moving on. Do not put hosts into maintenance mode with full data evacuation at 70% just for this change; that adds unnecessary migration load. If any host fails the large-ping test, stop and roll that host back to 1500. Keep the vMotion and vSAN networks separate and document the rollback step for each host.
3. After remediation, what monitoring proves stability and what should be documented?
Read the worked answer
Require vSAN health to be clear of MTU mismatch and partition alarms for a full observation period, successful 8972-byte vmkping from every host to every peer, resync and rebalance traffic back to normal, and no rising NIC drops or errors. Check VM latency and provisioning time. Document the before and after MTU table, switch ports, and the change record. Update the network diagram and add an MTU consistency check to the switch firmware upgrade runbook so the next firmware event does not repeat the mismatch.