Brownfield Consolidation for a 48-Rack Hall with an 8 kW per Rack Limit
A retailer is consolidating two aging data centers into one hall with 48 racks. Each rack is limited to 8 kW and 42 RU, and the compute refresh in Q3 will use 20 servers per rack, each with 2×25GbE. East-west traffic is 78% of the total and is forecast to grow 40% per year for three years. The facilities team can provide only 12 fiber pairs from the hall to a core room that has 4 RU of rack space. The business wants leaf-to-spine oversubscription no worse than 3:1 today and 4:1 after refresh, with one logical fabric that can extend to a second hall. The incumbent design is three-tier access/aggregation/core with STP and VRRP. The CIO asks whether to keep the three-tier model or move to spine-leaf, and how to handle the fiber limit.
1. Should the spines be placed in the core room as originally planned? Justify the topology using the fiber and space constraints.
Read the worked answer
No. A leaf with 4×100G uplinks needs four duplex fiber pairs to the spine layer, so 48 leaves require 192 pairs, not 12. Even a single uplink per leaf would need 48 pairs. The 4 RU core-room allowance is also too small for six or more 1 RU spines plus patch panels and cable management. The workable design is to put four to six spines in the hall and use the 12 fiber pairs as 12×400G (or 12×100G) uplinks from the hall spines to a redundant core border or super-spine pair. Alternatively, negotiate at least 192 pairs for a spine-in-core design. Three-tier with STP is a poor fit because 78% east-west traffic and 40% annual growth would force traffic through aggregation and root bridges, adding latency and blocking links. Spine-leaf in the hall keeps east-west traffic local and leaves the 12 pairs for inter-hall extension.
2. Calculate leaf-to-spine oversubscription for a 48-port 25G leaf with 4×100G uplinks serving 20 dual-25G servers. Does it meet the requirements?
Read the worked answer
Access bandwidth is 20 servers × 2 × 25G = 1,000G. Uplink bandwidth is 4 × 100G = 400G. The ratio is 1,000/400 = 2.5:1, which meets both the 3:1 today and 4:1 after refresh targets. The leaf uses 40 of its 48 25G ports, leaving 8 spare for growth or a break-out. The trap is to size the leaf for a fully populated 48-port rack: 48 × 2 × 25G = 2,400G, giving 2,400/400 = 6:1, which fails. The rack power limit, not the switch port count, controls server density. If the business later allows more than 20 servers per rack, the design must add uplinks or use 6×100G (600G, 4:1 at 48 servers) or 8×100G. Document the 20-server rack rule as a design constraint, not an assumption.
3. How should this fabric extend to a second hall later without stretching Layer 2 across the halls?
Read the worked answer
Keep each hall as an independent underlay and use EVPN-VXLAN over the 12-pair inter-hall link for Layer 2 and Layer 3 tenant extension. Each hall gets its own spine layer and its own eBGP underlay; the inter-hall link is a DCI between border or super-spine devices with unique ASNs. Advertise tenant subnets as EVPN Type-5 routes and use distributed anycast gateways in each hall so hosts always have a local default gateway. Do not extend STP or run a single large Layer 2 domain across the DCI; that would reintroduce blocked links and slow convergence. Use ESI-LAG for dual-homed servers within a hall, and route between halls. Plan the 12 pairs as a redundant DCI (for example, six pairs per border pair) and verify that the DCI capacity meets the projected 40% east-west growth before the second hall opens.