Intermittent VM Outages After Adding a Second Uplink
Four ESXi hosts connect to a pair of top-of-rack switches that form a vPC domain, with each host having two 10 Gb adapters, one cabled to each ToR. In the original design only one adapter per host was active. To double bandwidth, an engineer moves both adapters into the same vSwitch, selects 'Route based on IP hash,' and configures the host-facing ports on each ToR as a static (mode on) port-channel: switch A gets all four hosts' links in one channel, and switch B builds its own separate channel. No LACP is configured. The vPC peer-link exists, but the host ports are not vPC members. Within a day, switch logs show the same VM MAC addresses flapping between the two ToRs, two VMs lose connectivity for 30 to 45 seconds during vMotion, and a VLAN that exists only on switch A is unreachable from half the hosts. The vSwitch MTU is 9000; the ToR uplinks to the core are 1500, and jumbo is configured only on the peer-link.
1. Explain the MAC flapping and the 30 to 45 second connectivity losses. What is wrong with the port-channel design?
Read the worked answer
The two static port-channels are independent from the fabric's point of view, and the ToRs are joined by the vPC peer-link. A broadcast or unknown-unicast frame from a VM can enter switch A, cross the peer-link, and egress switch B's port-channel back toward the same host, which is a loop between the two ToRs. MAC learning then flaps because frames for the same VM arrive both on the local port-channel and over the peer-link, and each move triggers relearning. The 30 to 45 second losses are that reconvergence plus the port-channel hash landing vMotion traffic on a path that is momentarily black-holed. Root cause: a host-facing port-channel that spans two vPC peers must be a vPC on those peers, using the same vPC number on both, so the pair presents one logical channel. Use LACP active on the vSwitch team and mode active on the vPC legs rather than static mode on, which has no negotiation and no misconfiguration protection. If vPC is not available on the host ports, do not team with IP hash: use route based on originating virtual port with active/standby uplinks and no switch-side channel.
2. Why does one VLAN reach only half the hosts, and how do you fix it?
Read the worked answer
IP hash distributes each flow across all uplinks in the team, so every uplink must carry an identical VLAN set. If a VLAN exists only on switch A, frames for it arriving at switch B are hashed onto a link and dropped, or land on a host whose active path is switch A, which is exactly the 'half the hosts' pattern. The fix is consistency, not a workaround: allow that VLAN on both vPC legs for all four hosts, and make sure the vPC peer-link carries every VLAN the vPCs use, since a VLAN missing from the peer-link will fail to traverse the pair for traffic that has to cross. Check for native VLAN mismatch and VLAN pruning on the uplinks at the same time, because pruning that removes the VLAN from one side recreates the same symptom. Verify from each host, not from one: confirm the VMkernel and VM port groups see the VLAN on both uplinks, and validate with a ping from a VM on each host to a gateway in that subnet before declaring it fixed.
3. Assess the MTU configuration on this design and describe how you would validate it.
Read the worked answer
The design has a classic partial-jumbo path: the vSwitch is at 9000, the peer-link is at 9000, but the uplinks to the core are at 1500. Any large frame generated by a VM or by vMotion will be forwarded until it reaches the first 1500 hop, where it is dropped, and because the sender never learns about it, the symptom is a hang or stall rather than an error. That matches vMotion appearing to stall near completion. Path MTU has to be uniform end to end, or the endpoints must use the smallest value on the path. Choose one: either enable 9000 consistently on the vSwitch, both vPC legs, peer-link, uplinks, and the core, or bring the vSwitch back to 1500. Validate with a do-not-fragment ping from each VMkernel interface, for example ping with the DF bit set and a payload of 8972 bytes for a 9000 MTU, and step the size down until it succeeds to find the real path MTU. Also confirm that jumbo is configured on the vMotion VMkernel and not just on the vSwitch, and that vMotion is on a dedicated VLAN so a mismatch cannot silently affect production traffic.