NCA-GENL: Intermittent VLAN 40 Failures on a balance-slb Bond

Intermittent VLAN 40 Failures on a balance-slb Bond

A three-node AHV cluster runs AOS 6.7 with two 10 GbE uplinks per host (eth0, eth1) in the default OVS bond br0-up, configured as balance-slb. eth0 connects to switch A and eth1 to switch B; the switches are not stacked and have no MLAG. Storage and guest traffic share the bond. A new application is placed on VLAN 40, which is trunked on both switch ports. After go-live, users report that some VLAN 40 VMs work normally while others time out on roughly one in three connections to the app; the affected VMs can ping the gateway but large file transfers stall. VLAN 10 VMs on the same hosts are unaffected. A switch firmware upgrade was completed the previous weekend, and the network team confirms 'VLAN 40 exists on both switches'. The application owner wants the VMs moved to a different host, and someone suggests rebuilding the bond as LACP.

1. What is the most likely cause, and what evidence would confirm it before anyone changes configuration?

Read the worked answer

Most likely cause: balance-slb assigns each VM's flows to one uplink using a hash (based on source MAC and VLAN), not per packet. If VLAN 40 is missing or pruned from the allowed list on one of the two uplink ports - a classic casualty of a firmware upgrade that resets trunk allowed-VLAN lists - every VM hashed to that uplink loses VLAN 40 connectivity while VMs hashed to the other uplink are fine. That matches the symptom pattern exactly: some VMs fail, others do not, and VLAN 10 is unaffected because its allowed list was preserved.

Evidence to gather:

On a CVM: ovs-appctl bond/show br0-up to see the SLB slaves and which MACs are pinned to each; map the failing VMs' MACs to a slave.
manage_ovs show_uplinks and ovs-vsctl show to confirm bond mode, VLAN configuration on br0, and that both interfaces are members.
On both switches: the actual trunk allowed-VLAN list per port, the native VLAN, and the MAC address table for a failing VM's MAC.
A quick test: ping the gateway from a failing VM while watching which physical port the switch learns its MAC on.

Confirm before touching the bond. Changing bond mode on a hunch converts a one-port config error into a cluster-wide outage.

2. Is moving the affected VMs to another host a valid workaround? Explain what it does and does not change.

Read the worked answer

It is not a fix. Moving a VM to another AHV host re-runs the hash against that host's uplinks, so the VM may land on a working link and appear to recover. Three problems:

The fault is in switch port configuration, not in host placement. Any VM that rehashes onto the same broken uplink path on its new host fails again, and the failure reappears after the next bond rebalance or host reboot.
Live migration itself depends on the storage and management networks using the same bond. If the broken link carries management traffic for that host, the migration may hang or fail, widening the outage.
It hides the fault from monitoring. The next application deployed on the affected host inherits the same latent defect.

Use migration only as an emergency mitigation while the switch port is corrected, and record why. The real fix is restoring VLAN 40 to the allowed list on the affected uplink port, then verifying with per-slave connectivity tests from each host.

3. Would converting the bond to LACP improve this design? State the prerequisites and the trade-offs on this specific topology.

Read the worked answer

On this topology, LACP is not automatically better and may break things.

LACP requires the switch side to be configured as a port-channel with matching LACP parameters. Here eth0 goes to switch A and eth1 to switch B, which are unstacked. Without MLAG or a stack, each switch sees a single-member link, so you get two independent one-link bundles at best - no aggregated throughput - and mismatched LACP settings produce an interface that will not pass traffic.

Trade-offs:

balance-slb needs no switch-side bonding, survives a single switch failure, and hashes per VM flow. Its weakness is exactly this scenario: a silent per-port VLAN or STP problem affects only the VMs pinned to that link, which is hard to spot.
LACP gives standard link-failure detection and a well-understood switch-side model, but only if the switches are stacked or MLAG'd into one logical device, and it does not give better per-flow balancing than SLB.

Recommendation: keep balance-slb with the current two-switch design and fix the trunk. If the network team will stack or MLAG the switches, revisit LACP as a separate, tested change.