Data Center and Cloud Connectivity Design
This domain covers fabric topology and oversubscription, overlay control planes, storage transport, hybrid cloud attachment, and the cost and failure models unique to each.
Spine–leaf and oversubscription
A Clos fabric connects every leaf to every spine with ECMP and no spanning tree. Design is dominated by oversubscription: sum the southbound (server-facing) bandwidth and the northbound (spine-facing) bandwidth and express the ratio consistently, uni-directionally. Classic guidance is 1:1 to 3:1 at access and higher at the border, but modern east-west-heavy applications often justify closer to 1:1 because server-to-server and storage flows dominate.
Overlay and control plane
VXLAN with a BGP EVPN control plane provides MAC/IP distribution without flood-and-learn; the underlay carries loopbacks and point-to-point links routed by eBGP or OSPF. Distributed anycast gateways let every leaf serve the same default gateway MAC/IP, so server mobility needs no first-hop change. Server dual-homing uses vPC/MLAG or EVPN multihoming; hosts should not depend on STP. Underlay eBGP commonly gives each leaf a unique ASN, or one ASN with allowas-in when the ASN is shared.
Storage
Fibre Channel remains the deterministic option; iSCSI rides the IP fabric; RoCEv2 and NVMe-oF require a lossless class using PFC and ECN marking with careful buffer tuning - a misconfigured PFC can create head-of-line pause storms that stall unrelated traffic.
Cloud attachment
Compare Direct Connect/ExpressRoute against site-to-site VPN on bandwidth, latency consistency, lead time and cost. Private VIF/peering uses BGP with either a private or public ASN, one VLAN per virtual interface. Inside the cloud, a hub VPC or transit gateway centralizes inspection and routing; watch cross-AZ and egress charges, which frequently exceed the design's bandwidth assumptions. Place latency-sensitive tiers in the region nearest users and keep chatty tiers in one zone.
Worked example
A leaf has 48 × 25 Gbps downlinks (1.2 Tbps) and 8 × 100 Gbps uplinks (800 Gbps) - 1.5:1, which is fine for general compute. The storage team states that array-to-server replication is 40% of traffic and must not be oversubscribed. At 1.5:1 the storage flows can be dropped under congestion, so either add uplinks to reach 1.2 Tbps northbound or land the storage ports on a dedicated leaf pair. Choose dedicated storage leaves and a separate lossless class rather than making the whole fabric lossless, which risks pause storms affecting all traffic.
Traps that cost marks
- Designing the fabric around spanning tree or expecting STP to protect VXLAN loop paths.
- Mixing unidirectional and bidirectional numbers in oversubscription claims.
- Ignoring cloud egress and cross-AZ costs, and assuming VPN-over-Internet meets voice jitter budgets.
- Enabling PFC without ECN and buffer tuning, creating pause-storm propagation.