Two-Spine Data Centre Refresh Under an Optics Budget
Meridian Payments is refreshing a 36-rack data centre that will hold 288 dual-attached servers (8 per rack, 2 × 25 GbE each), an all-flash array with 8 × 100 GbE ports, and 80% east-west traffic between application tiers. The brief caps the spine layer at exactly two switches because of rack space and optics budget, targets 3:1 oversubscription at the leaf, and requires that a single spine failure still leaves at least 2:1. The current fabric is a vPC pair of 10 GbE switches running STP, with VLANs stretched to every rack and 4:1 oversubscription. Operations will not accept a control plane they cannot troubleshoot with OSPF. The virtualization team wants live migration to work across any two racks in the same pod, and storage intends to run RoCEv2. Floor space and power are fixed; the optics line item is the binding constraint on the whole project.
1. Compute the leaf and spine quantities and uplink speed needed to meet 3:1 and survive a spine failure, and state the recommendation.
Read the worked answer
Total server-facing ports: 288 servers × 2 NICs = 576 × 25G. A 48-port 25G leaf used at 32 ports serves two racks (16 servers, 32 ports), so you need 18 leaves, each with 800 Gbps of downlink (32 × 25G). Meeting 3:1 requires at least 267 Gbps of uplink per leaf; 4 × 100G gives 400 Gbps (2:1), and 2 × 400G gives 800 Gbps (1:1). Now test the two-spine cap. Eighteen leaves × 4 × 100G needs 72 spine ports, so two 32-port 100G spines (64 ports) cannot carry it - the brief is arithmetically impossible with 100G uplinks. Eighteen leaves × 2 × 400G needs 36 ports, which fits two 32-port 400G spines with 28 ports spare. Recommendation: 18 leaves, 2 × 400G uplinks per leaf, two 400G spines. Steady state is 1:1, well beyond the 3:1 target, and losing one spine leaves 400 Gbps per leaf, exactly 2:1 - which is why this meets both conditions simultaneously. If 400G optics cannot be funded, the honest alternatives are three 48-port 100G spines (breaking the cap) or accepting 16 leaves and 512 ports, which does not cover 576 server ports. Say that out loud rather than quietly oversubscribing.
2. Decide the Layer 2/Layer 3 boundary and explain how live migration across any two racks in a pod is achieved without stretched VLANs everywhere.
Read the worked answer
Route to the leaf. Every leaf gets a loopback, a /31 to each spine, and OSPF with passive interfaces toward servers, which satisfies the operations requirement and gives fast, well-understood convergence. Servers get a small per-leaf subnet rather than a rack-spanning VLAN, and the VLAN sprawl that currently reaches all 36 racks disappears. For live migration, the constraint is that the guest must keep its IP address when it moves, which means the same subnet must exist on both source and destination leaves. Do not solve this by stretching VLANs fabric-wide; solve it at pod scope. Define pods of two adjacent racks (which map exactly to one leaf pair under the 2-racks-per-leaf plan) and extend only that subnet across that pair, or better, use VXLAN with a distributed anycast gateway so the same gateway IP and subnet exist on every leaf and the ARP/ND tables are suppressed. Migration then works across any two racks in the pod, and the Layer 2 failure domain is bounded to two racks instead of 36. Keep the pod boundary a design decision, not a vMotion accident: if the virtualization team wants cross-pod migration, that is a new requirement and a new conversation about overlay scope.
3. Recommend how to attach the all-flash array and run RoCEv2 without wrecking the fabric's failure behaviour.
Read the worked answer
RoCEv2 needs a lossless class, which means PFC and ECN configured consistently end to end; a single hop without it produces silent drops and the array retransmits, which looks like an application problem and is very hard to find. On a converged fabric that also carries ordinary TCP, PFC headroom, buffer sizing, and the risk of pause storms make this fragile. Dual-home the array's eight 100G ports across two leaves - not one - so losing a leaf does not take storage down, and put storage on its own VLAN with a dedicated lossless priority, jumbo MTU (4200) verified end to end including the server NICs, and PFC enabled only on the array-facing and inter-leaf paths where the storage traffic actually flows. Verify that the spine-failure case still holds 2:1 for storage: at 400 Gbps per leaf in degraded state, a single array port pair is fine, but confirm the lossless queue's guaranteed bandwidth is provisioned for it rather than left to best effort. Finally, test with real workloads - many RoCE designs pass a ping test and fail under a synthetic incast pattern. If the operations team cannot own PFC and ECN tuning, the cheaper answer is Fibre Channel or NVMe/TCP for the array, and that trade should be made explicitly, not by default.