Intermittent Guest Connectivity After Rack Expansion
A four-node hyperconverged cluster is split across two racks. Each host has two 10 GbE NICs in an active/backup bond, one link to ToR-A and one to ToR-B. The switches are not MLAG; a single interswitch link is STP-blocked. VLAN 100 is management, VLAN 200 is storage with MTU 9000, and VLAN 300 is guest with MTU 1500. A fifth node is added in rack B. Its CVM joins, but during a live migration test, pings to the guest VM drop for 20-40 seconds. Storage latency alerts appear on the new node. From the new node, ping -M do -s 8972 to an existing CVM fails; between existing nodes it succeeds. The new switch port is a trunk with native VLAN 1 and allowed VLANs 100,200,300; existing ports use native VLAN 100. Cluster health shows the new CVM as degraded. No broadcast storm is reported.
1. Which faults explain the jumbo-ping failure and the storage latency? What would you check first?
Read the worked answer
The jumbo-ping failure is a path-MTU problem, not a routing problem. The new node can reach the CVM, but 8972-byte packets with the DF bit set fail, which means at least one segment on VLAN 200 cannot carry 9000-byte frames. The most likely cause is that the new switch port or its VLAN 200 interface is still at MTU 1500. Storage replication and CVM communication rely on jumbo frames; when large frames are dropped, TCP retransmits or fragments, which shows up as storage latency and a degraded CVM.
Check in this order: (1) the physical switch port MTU on the new node's uplink; (2) the VLAN 200 SVI MTU on both ToR switches; (3) the host bridge and storage interface MTU; (4) end-to-end jumbo ping from every node to every CVM. Also verify the native VLAN mismatch, because native VLAN 1 versus 100 can break untagged control traffic and management, but it does not by itself explain a jumbo-frame failure on tagged VLAN 200. Fix the MTU first, then re-test migration and storage latency.
2. Why would guest pings drop for 20-40 seconds during live migration, and how would you verify the cause?
Read the worked answer
A 20-40 second drop during migration points to Layer 2 reconvergence or MAC-learning disruption, not to a storage outage. When the VM moves to the new host, its MAC address appears on a different switch port. Because the two ToR switches are not MLAG and the interswitch link is STP-blocked, traffic for that MAC may temporarily follow a stale path or wait for STP/MAC-table convergence. The native VLAN mismatch adds control-plane inconsistency and can make the problem intermittent.
Verify with a continuous ping and a simultaneous arp -a/MAC-table check on both switches. Record the VM's MAC before and after migration, then check which port learns it and whether a topology-change trap appears. Check STP topology-change counters and the interswitch link state. Also confirm whether gratuitous ARP or RARP from the migrated VM is being filtered. If the MAC move consistently triggers STP churn, the design fix is MLAG/vPC (or a supported multichassis LAG) and consistent native VLANs, not just a longer ARP timeout.
3. What design changes would prevent recurrence, and what should be validated before the next node is added?
Read the worked answer
The durable fix is to remove the Layer 2 asymmetry. Use MLAG or an equivalent multichassis LAG so both ToR switches share a consistent forwarding view, and standardize the native VLAN on every trunk. Keep storage and guest VLANs consistently tagged. Make MTU 9000 a documented standard for VLAN 200 on hosts, switch ports, and SVIs. If MLAG is not available, avoid pretending the pair is a single logical switch; document the active/backup behavior and test failover explicitly.
Before adding another node, validate: end-to-end jumbo ping in both directions, trunk allowed/native VLAN match, STP topology stability during a test migration, CVM health, and a live-migration ping test with zero or near-zero loss. Also capture switch logs during the test. The key judgement is that the jumbo-ping failure and the migration drop are separate faults with a shared root cause: inconsistent Layer 2 configuration across the two racks.