I85PREP
⌘ K
VCP-DCV study materials
VCP-DCV study resources

vMotion Crawls After the 25 GbE Refresh

VCP-DCV · Put your knowledge to work in context.

All case studies

vMotion Crawls After the 25 GbE Refresh

After a network refresh, a 4-host cluster runs a single vSphere Distributed Switch with two 25 GbE uplinks per host in one LAG, negotiated with LACP to a top-of-rack MLAG pair. The VDS is set to MTU 9000. vSAN, vMotion and guest traffic all share those uplinks, on separate distributed port groups, with Network I/O Control enabled at default shares. On three hosts, vMotion of a 64 GB VM completes in a couple of minutes. On esx04 the same VM failed twice with a migration timeout, and the one successful attempt took 3 hours 40 minutes. vSAN resynchronisation on esx04 is also consistently the slowest in the cluster. Drivers and firmware are identical on all four hosts.

1. Before touching any vMotion or DRS setting, which single configuration check best explains a host-specific, order-of-magnitude throughput difference - and how would you prove it rather than suspect it?

Read the worked answer

The candidate: an MTU mismatch on esx04's path

Order of magnitude is the clue. A missing LAG member or a degraded link costs 2x, not 50x. A path that negotiates 1500 bytes under a 9000-byte expectation costs everything, because large frames are dropped and TCP falls back to retransmission and tiny segments.

Prove it

  1. From esx04's vMotion VMkernel, run a large-frame probe with the don't-fragment bit to a VMkernel on a healthy host: vmkping -d -s 8972 <peer vmkernel>. Repeat from a healthy host.
    • Healthy host succeeds; esx04 fails or reports fragmentation needed.
  2. Compare with a 1472-byte probe from esx04. If small frames succeed and large frames fail, it is path MTU, not routing or a down link.
  3. Then verify the physical side: the switch port(s) carrying esx04's LAG members, the MLAG peer link, and the LACP port-channel MTU. In refreshes the host side is templated and the switch port is configured by hand - this is exactly the port that gets missed.
  4. Check the LAG membership from esxcli network vswitch dvs vmware lacp status to confirm both uplinks are actually up and bundled, so you can rule out the 2x case cleanly.

Why it fits the vSAN symptom too

vSAN and vMotion share the same uplinks. If esx04's path cannot carry 9000-byte frames, its vSAN traffic degrades identically - one root cause, two symptoms. vSAN also requires uniform MTU across all participants, so a single misconfigured host degrades the whole cluster's data path, not just its own.

2. Once MTU is corrected, the team proposes dedicated uplinks for vMotion. Do the arithmetic and give a recommendation.

Read the worked answer

Arithmetic

  • A 64 GB VM is roughly 69 GB of memory to transfer, plus page tables and a final delta pass.
  • One 25 Gb/s link at line rate is ~3.1 GB/s, so ~22 seconds in theory.
  • Real vMotion achieves perhaps 50-70% of line rate on a large VM with a single flow: ~35-60 seconds.
  • The observed 3 h 40 m is ~13,200 seconds for 69 GB, about 5 MB/s - roughly 0.2% of a single link. Nothing about that is a bandwidth sizing problem.

LACP detail that matters

vMotion is one TCP flow. Under IP-hash teaming that flow is pinned to a single LAG member, so the ceiling for any single migration is 25 Gb/s regardless of how many uplinks are bundled. Dedicated uplinks therefore do not raise the per-migration ceiling.

Recommendation

Do not buy dedicated uplinks for throughput - after the MTU fix you are already near the single-flow ceiling. If large-VM migrations must be faster, use multiple vMotion VMkernel adapters (multi-NIC vMotion), each bound to a different uplink, so ESXi can spread a migration across flows. Consider dedicated uplinks only for isolation reasons - keeping vMotion stun traffic away from vSAN resync, or applying strict shares - and justify that on operational grounds, not on a bandwidth calculation that does not hold.

3. During a vSAN rebuild, vMotion times out and the team asks whether NIOC should be trusted to protect vMotion. What does NIOC actually guarantee on this design, and what would you configure?

Read the worked answer

What NIOC does and does not do

  • NIOC allocates by shares, reservations and limits across system traffic types (vSAN, vMotion, guest, management) on the same set of uplinks. Shares only matter under contention, which is exactly the rebuild scenario - so enabling it is correct but shares alone are best-effort.
  • A reservation is a hard floor, but only up to the physical uplink capacity, and only for traffic types inside the switch. It cannot reserve bandwidth on the MLAG peer link or upstream.
  • Therefore NIOC cannot fix an upstream bottleneck, a saturated ToR uplink, or an MTU black hole. Confirm the bottleneck location with per-uplink throughput counters before promising anything.

What to configure

  1. Give vMotion and vSAN explicit reservations sized to the traffic you must guarantee, with shares above guest traffic; leave guest traffic on shares only.
  2. Separate vSAN resync from the failure: vSAN rebuild throttling is controlled at the vSAN layer, not by NIOC. Cap resync during business hours so a single disk or host loss cannot consume the whole uplink budget.
  3. Schedule bulk vMotion outside rebuild windows, and use DRS maintenance-mode evacuations rather than ad-hoc bulk migrations.
  4. Re-test after the MTU fix: if per-uplink counters show plenty of headroom during rebuilds, the timeouts were transport errors, not contention, and tuning NIOC would have masked the real fault.