Colo Consolidation: Spine-Leaf Under Power and Fiber Limits

Start with the constraints, not the vendor preference. The inter-row fiber cap is 24 strands. With dual-strand links, that is 12 inter-row links total. A full spine-leaf design with four spines in row A and every leaf in row B dual-homed would consume 4 spines × 2 links × 2 strands = 16 strands, leaving only 8 strands for growth; six spines would consume all 24. Power also matters: a 6 kW rack holding a 48-port 25G leaf at ~450 W plus servers is feasible, but a spine at ~800 W in the same rack reduces server density. The current 4:1 oversubscription is the real pain: 40 servers × dual 25G = 2 Tb/s per rack; eight 100G uplinks provide 800 Gb/s, or 2.5:1, a clear improvement. A pod-based design (four pods, two spines per pod) uses fewer inter-row links and keeps power local, but adds an inter-pod tier and more latency for the 82% east-west traffic. My recommendation is a phased pod-based leaf-spine with a super-spine reserved for later: it fits the fiber budget, reduces oversubscription below 3:1 within a pod, and preserves a migration path. Full spine-leaf is technically superior but not deployable within the 24-strand and 6 kW constraints without renegotiating facilities.

Treat the migration as a series of bounded L2/L3 domains, not a single cutover. Build the new fabric alongside the VSS core and make the old core a border for un-migrated VLANs. For each wave, dual-attach servers with LACP to a leaf pair, keep the old path passive, and move the default gateway last. Use a temporary border leaf with a VRF to hand off between old and new; avoid stretching VLANs across both fabrics, because that reintroduces spanning tree and creates asymmetric flooding. Move the analytics cluster first because it is latency-sensitive but has a small blast radius, then backup, then tier-1 applications. Schedule one rack per window and require a rollback that is just moving the LACP active link back. Do not use VSS as a transit between fabrics; route at the border. Freeze VLAN additions during each wave and pre-stage anycast gateway and ARP suppression. The 30-day move-in penalty argues for starting with two racks, not for a big-bang.

Require measurable gates, not a feeling. First, p99 intra-fabric latency below 5 ms at 70% load and no sustained buffer drops on any spine egress queue; the 40–60 ms backup spike must be gone. Second, oversubscription no worse than 3:1 at the leaf and BUM traffic below 2% of total; above that, ingress replication and ARP flooding will mask themselves as latency. Third, routing and overlay convergence under a single link failure within 1 second, with no MAC mobility events above baseline, and ARP/ND cache below 70%. I would also require a 24-hour soak with production-like backup traffic and a verified rollback that returns the rack to the old core in under 15 minutes. If any gate fails, delay the next wave rather than widen the window.