ECMP Hashing and a Single Hot Flow Across the Spine
A leaf-spine fabric has 4 spines and 16 leaves, all 100G, with ECMP across the spines (4-way per leaf). A new analytics workload runs a single large TCP flow from a source on leaf-03 to a destination on leaf-11. Monitoring shows the flow pinned to one spine at ~95 Gbps while the other three spines carry ~30 Gbps each from other traffic. The application team reports the transfer takes far longer than the 100G line rate should allow and asks the network team to "add more bandwidth." The fabric has spare capacity; the bottleneck is not the spines overall. The flow is a single TCP connection that cannot be split.
1. Explain why adding more spines or more bandwidth will not speed up this specific flow, and what limit the flow is actually hitting.
Read the worked answer
ECMP is per-flow, not per-packet. A single TCP connection hashes to exactly one next-hop (one spine) for its lifetime, so it is capped at the bandwidth of that single path - roughly one 100G link, and in practice less after overhead and any per-link contention. The other three spines are irrelevant to this flow. Adding a fifth spine changes the hash space but still gives this flow exactly one path. The flow is hitting the single-path bandwidth ceiling, not aggregate fabric capacity. That is why the transfer is slow despite idle spines: the fabric is fine; the flow cannot use it. "Add bandwidth" would only help if it increased the per-path rate (e.g., 100G→400G links) or if the flow could be parallelized.
2. Give at least two practical mitigations, distinguishing what the network team can do from what requires application changes.
Read the worked answer
Network-side (limited):
- Increase per-path link speed (e.g., 400G spine links) so the single hashed path is wider - expensive and only helps this class of flow.
- Tune ECMP hash inputs to spread different flows better, but this does not help a single flow; be honest that hashing cannot split one TCP connection.
- Ensure no per-link congestion (e.g., other elephant flows) is stealing capacity from the chosen spine; QoS or flow-aware placement can reduce contention on that path.
Application-side (the real fix):
- Parallelize the transfer into multiple TCP connections/streams (e.g., multi-threaded transfer,
gridftp-style, or object-store multipart upload). N parallel flows can hash across N spines and approach aggregate bandwidth. - Use a protocol or tool designed for high-bandwidth WAN/LAN transfer that opens multiple streams.
The correct message to the app team: the fabric has headroom; the single-flow design is the constraint. The durable fix is parallelization, with link-speed upgrades as a costly fallback.