Choosing an underlay and MTU plan for a 4-spine/12-leaf refresh
A retailer is refreshing a 2016-era pod with 4 Aruba CX 8360 spines and 12 CX 8325 leaves: 100G fabric links, 25G host ports, two racks per leaf pair, servers dual-homed with LACP. The team runs OSPF today, has no BGP production experience, and wants sub-second reconvergence plus a fabric that is fully reproducible from Ansible. Hosts use a 1500-byte MTU, and a legacy inline IPS that only supports 1500 bytes will attach to a border leaf. Target oversubscription is 3:1 at the leaf. Three questions remain open in the design review: which underlay routing protocol to standardise on, what MTU plan to adopt, and whether to keep VSX or move to EVPN multihoming for server attachment.
1. Which underlay routing protocol should the fabric standardise on, and what conditions would change your answer?
Read the worked answer
Criteria that actually decide this: convergence target, ECMP behaviour, failure blast radius, team skill, and how much of the config can be templated with confidence.
- OSPF (single area): any link change floods LSAs to all 12 leaves. With BFD and tuned hello/dead timers you can still hit sub-second reconvergence, and the team already operates it. Cost: fabric-wide blast radius and an LSDB that is harder to reason about when an automated push goes wrong.
- eBGP (one ASN per leaf): one session per spine per leaf, BFD for fast failure detection, deterministic best-path with an explicit
maximum-paths, and no flooding - a flapping host-facing link touches nothing else. Cost: retraining, and BGP requires discipline in the templates (router-IDs, ASN scheme, no accidental default-originate).
Recommendation: eBGP for the spine-leaf layer if training is funded in the same project - it is the better fit for a fully templated fabric and matches where the overlay is going. If training is not funded, OSPF area 0 with BFD is a legitimate, working answer at this scale; do not treat BGP as mandatory at 16 switches. What would flip the decision: an existing BGP-free operations model with no training budget, or requirements for a single protocol across a much larger multi-pod topology.
Either way: address links with /31s (verify whether your AOS-CX release supports BGP unnumbered before relying on it), set maximum-paths explicitly, keep L2 gateway functions on VSX SVIs, and prove convergence by pulling a spine uplink under traffic rather than trusting a design document.
2. Design the MTU plan end to end, including the 1500-byte IPS, and describe how you would validate it before cutover.
Read the worked answer
The arithmetic: VXLAN encapsulation adds roughly 50 bytes to the inner frame (8 VXLAN + 8 UDP + 20 outer IP + 14 outer Ethernet). A 1500-byte inner payload becomes a ~1564-byte frame on the wire. If any underlay hop is at 1500, the outer packet is dropped (DF is effectively set), and it fails only for large frames - pings and small API calls succeed while bulk transfers stall. That symptom pattern is the tell.
Plan:
- Set IP MTU 9000 (or 9216 if your hardware and templates support it consistently) on every spine-leaf link and every VTEP-facing routed interface. Standardise on one value; mixed MTUs are the single most common cause of "mysterious" large-frame loss.
- Keep host-facing ports at 1500. The VTEP does not fragment - never design around outer fragmentation.
- The IPS: it cannot terminate a VXLAN tunnel and cannot see a 9000-byte frame. Place it in a routed, non-VXLAN path (service leaf with an L3 handoff at 1500), or ensure only routed, already-decapsulated traffic reaches it. If the IPS must sit inline in a stretched segment, large frames will fail and no amount of ACL tuning will fix it.
Validation before cutover: DF-bit ping sweeps at 1472 / 1500 / 1600 / 2000 / 8972 between every leaf loopback pair and across each VRF; a cliff between 1472 and 1600 means a 1500-byte hop. Confirm the actual configured MTU per port rather than assuming the template applied, and check VXLAN tunnel counters for drops after the sweep.
3. The team already runs VSX for server attachment. Should the refresh move to EVPN multihoming at the same time?
Read the worked answer
VSX: two control planes joined by ISL and keepalive, active-active LACP, active-gateway for a consistent anycast MAC. Mature, understood by the team, works with the servers' existing LACP bundles. Costs: the ISL carries all orphan and synchronised traffic and is a shared failure domain, split-brain handling must be documented, and upgrades are coordinated pair events.
EVPN-MH: ESI-based, no ISL, standards-based, converges with the overlay you are about to build. Costs: it depends on a healthy EVPN underlay/overlay, requires careful ESI and designated-forwarder configuration, and changes the troubleshooting model entirely.
Judgement: do not change two foundational layers in one window. Since the overlay is not deployed yet, keep VSX for this refresh - it delivers the LACP active-active requirement today with skills the team has - and schedule EVPN-MH as a defined phase 2 once the overlay has run clean for a quarter. The exception: if the overlay and the leaf refresh land in the same change window, build EVPN-MH from day one, because running VSX now means a second server-facing migration later.
Whichever you choose, write the split-brain and orphan-port runbooks first; those are the operational decisions, not the protocol choice.