VCP-VVF Admin (2V0-16.25): vMotion Failing After an MTU Change on a Distributed Switch

vMotion Failing After an MTU Change on a Distributed Switch

Twelve hosts sit on a vSphere Distributed Switch 8.0 with two 25 GbE uplinks per host in an active/standby teaming policy (route based on originating virtual port). The physical switches are configured with MLAG and LACP, but no LAG exists on the vDS. NFS storage and vMotion share the same two uplinks on separate VLAN-backed distributed port groups. Last week the team raised the vDS MTU to 9000 to prepare for vSAN, and the physical switch ports were left at 1500. During a load test, vMotion of a 64 GB VM failed at 14% with 'Migration exceeded maximum switchover time', and one host logged network rollback events. NIOC is enabled with vMotion at 50 shares and NFS at 100 shares. vmkernel adapters for NFS and vMotion exist on every host.

1. Give your diagnosis of the vMotion failure and the exact commands you would run to confirm or reject it.

Read the worked answer

Most likely cause - an MTU mismatch. The vDS is set to 9000 while the physical switch ports are 1500. Frames above 1500 are dropped on egress to the physical network. vMotion's bulk transfer stalls, the switchover phase times out, and the host logs rollback events when it reverses the change.

Why not NIOC - shares only arbitrate under contention. The failure at 14% with a hard timeout and rollback points at loss, not bandwidth starvation.

Confirmation commands (run on each host, comparing the NFS and vMotion vmkernels)

vmkping -d -s 8972 -I vmk1 <peer vmk IP> - -d sets DF, -s 8972 is 9000 minus 28 bytes of headers. A failure here while vmkping -s 1472 succeeds is conclusive.
Test every pair you care about: host-to-host vMotion vmkernels, and host-to-NFS-array if jumbo frames are intended there.
esxcli network vswitch dvs vmware list to confirm the vDS MTU and which port groups/vmkernels are attached.
Check physical switch interface counters for giants/drops and confirm the port MTU.

Reasoning - small pings pass on a mismatched path; only a full-size DF ping proves end-to-end jumbo support.

2. In what order should MTU be corrected, and how do you validate before declaring success? Include the risk to running workloads.

Read the worked answer

Order

Change the physical switch ports and the L3 path (routers, if vMotion crosses subnets) to 9000 first. Raising the MTU on the network side breaks nothing that currently works at 1500.
Then raise the vDS uplink MTU / port group MTU so hosts can send and receive large frames.
If you must go the other way (rolling back), reverse it: lower the vDS first, then the physical switch - never leave the physical path smaller than the virtual path.

Risk

Changing MTU on a vDS affects all hosts using it and briefly impacts networking as uplinks re-apply. Do it in a maintenance window, one host at a time if possible, and never while a vMotion or storage migration is in flight.
Do not change the MTU tied to a LAG or NIOC configuration in the same window - one variable at a time.

Validation

Full-size DF pings between every pair of vmkernels on every host (vMotion pair, NFS pair).
Re-run the 64 GB vMotion test, then a storage vMotion, then a live migration across the two MLAG members to prove both uplinks.
Watch for giants/drops on the physical switch after the change.

Reasoning - consistent MTU end to end is the only correct state; a mixed path is a silent packet shredder.

3. The network team now wants LACP on the vDS. What has to change on the vSphere side, and what are the failure modes you would guard against?

Read the worked answer

What changes

Create a Link Aggregation Group on the vDS (New Link Aggregation Group), assign the correct number of uplinks per host (matching the switch port-channel), and set the LACP mode to active. Then move hosts' pnics into the LAG.
Migrate uplinks one host at a time from the standalone uplink ports to the LAG. Changing the teaming on a dvUplink is disruptive, so schedule per-host maintenance.

Failure modes to guard against

LACP negotiation failure shuts the links down. If the switch port-channel and the vDS LAG disagree on mode, hash, or member count, the host loses all uplinks. Verify the port-channel is already up on the switch side before assigning a second host.
Wrong hashing assumption. With LACP, the vDS offers IP hash as the load-balancing policy; do not expect originating-virtual-port semantics. Check whether the physical switch hash (L3/L4) matches what you select.
NFS/vSAN path behaviour. A single flow still maps to one uplink, so a LAG does not fix a single-flow throughput limit - for NFS, use multiple vmkernels or multipathing rather than assuming aggregation helps.
No rollback path. Keep one host on the old uplink configuration as a control, and have console/iLO access ready.

Reasoning - LACP improves aggregate throughput and per-host redundancy, but it converts a per-host change into a coordinated switch-plus-vSphere change with a hard failure mode; sequence accordingly.