MLAG Peer Link Saturation During Storage Rebuild
A two-switch MLAG pair (leaf-01/leaf-02) serves 24 servers across two 40G MLAG bonds each. A storage cluster rebuild begins after a drive failure; the affected server pair pushes ~38 Gbps sustained across its bond for 40 minutes. Operators see the MLAG peer link (single 100G port-channel) spike to 92% utilization, and several unrelated servers on other bonds report intermittent 1-3 ms latency spikes and occasional TCP retransmits. The peer link carries both the MLAG control VLAN and the data VLANs. No spanning-tree topology changes are logged. Uplinks to the spines show normal utilization (~45%). The team's first instinct is to add a second peer-link member.
1. Explain why the peer link saturates even though the affected server's traffic is dual-homed and load-balanced across both leaves. What is the most likely traffic pattern causing this?
Read the worked answer
MLAG is not a single logical forwarding domain across the peer link. Each leaf forwards independently; the peer link only carries traffic that must cross it. The classic case is a server whose bond hashing sends a flow toward the wrong leaf for the destination - for example, a storage rebuild stream from server A (primary on leaf-01) to server B (primary on leaf-02). If the hash pins the flow to leaf-01, leaf-01 must forward it over the peer link to reach leaf-02's local port. With 38 Gbps of sustained east-west storage traffic, a large fraction can end up hairpinned across the peer link. Uplink utilization staying at 45% confirms the traffic is not leaving the pair - it is crossing the peer link. This is a hashing/placement artifact, not a capacity failure of the MLAG design itself.
2. The team proposes adding a second 100G member to the peer link. Evaluate this fix and describe at least two alternatives that address the root cause.
Read the worked answer
Adding bandwidth treats the symptom. It will relieve the immediate congestion, but peer-link traffic is unpredictable and can re-saturate as workloads move; it also complicates the peer link (LACP hashing across members, more failure modes) without changing the topology that forces hairpinning.
Root-cause alternatives:
- Improve bond hashing / flow distribution. On EOS, tune
port-channel load-balance(e.g., include L4 ports, or usehashfields) so that east-west flows between the two storage servers distribute across both leaves rather than pinning to one. - Co-locate chatty pairs. Place the two storage servers' primary MLAG members on the same leaf so their traffic stays local and never crosses the peer link.
- Route instead of bridge. If the storage VLAN can be routed, ECMP across the spines gives the pair two independent paths (leaf-01→spine→leaf-02), removing dependence on the peer link entirely.
- QoS/priority. If the peer link must carry this traffic, apply congestion management so latency-sensitive server traffic is not starved by the bulk rebuild.
Recommended: (1) + (2) as the durable fix, with (4) as a safety net.