Designing the Refresh With Three Engineers and No Room for a Rework
A manufacturer is refreshing two data centres with USD 900,000 total capital. Each holds 240 VMs on a three-tier estate. North–south peaks at 4 Gbit/s, but east–west traffic from a new microservice application is growing 30% a year and now exceeds north–south at peak. Today the sites run Catalyst 6500-class cores with Spanning Tree, 1 GbE top-of-rack, and VLANs stretched between sites to support the standby tier. The infrastructure team is three engineers, none with production VXLAN, EVPN or BGP-EVPN experience; one writes basic Python. About 40% of workloads will move to a co-location facility in 24 months. Target life is five years. The CIO wants a design that survives two simultaneous rack failures and can be operated by the current team.
1. Spine-leaf or a refreshed collapsed core? Make a recommendation and state what evidence would flip it.
Read the worked answer
Recommend two-tier spine-leaf per site: 2 spines and 4 leaves, eBGP underlay, ECMP, every leaf dual-homed to both spines. Reason: east–west already exceeds north–south and is compounding at 30%; a collapsed core forces every east–west flow through one pair, and STP blocks half the links.
But spine-leaf alone does not satisfy the requirement. Two simultaneous rack failures is a compute placement problem, not a fabric problem. The design must spread each application's VMs across at least three racks with anti-affinity, and keep at least one spine and three leaves outside any two-rack failure domain. Write that into the design document or the CIO's requirement will not be met.
What would flip the decision: east–west growth flattening below 10% and staying there for two quarters; the co-location move being cancelled; or a capital ceiling near USD 400k, where a collapsed core with MC-LAG and 10 GbE ToR is defensible at 240 VMs. Also flip if the site cannot support the fibre count - spine-leaf triples the inter-rack cabling.
2. How would you handle workload mobility and segmentation - stretched VLANs, EVPN-VXLAN, or routed Layer 3? What has to be true before you deploy an overlay?
Read the worked answer
Default to routed Layer 3 to the leaf with no stretching. Most three-tier workloads do not need L2 adjacency; a standby tier with DNS/GSLB or a load balancer health check works fine across subnets. This removes the failure domain that today lets a broadcast storm in one site reach the other.
Use EVPN-VXLAN selectively, only for the workloads that need an any-rack subnet during a live migration, and cap the number of stretched subnets - every stretched subnet is a shared failure domain.
Prerequisites before overlay, in order:
- Underlay built and documented by someone who can troubleshoot BGP - training budget of roughly USD 10–15k per engineer plus a lab, or budget for a managed service.
- Automation for the underlay (IaC templates) so a 4-leaf change is not hand-typed at 02:00.
- A support contract that covers EVPN, and a documented rollback to routed L3.
If those are not funded, do not deploy the overlay - a fabric nobody can operate fails at 03:00, not in the design review.
3. Name the top three design risks and say how each is mitigated or explicitly accepted.
Read the worked answer
- Skills gap (highest). Three engineers, no EVPN experience, five-year life. Mitigate: training and lab before cutover, one contractor or partner for the first 90 days, runbooks validated by a game day. Accept residual risk in writing with the CIO if training is cut.
- Cabling and optics cost overrun. Spine-leaf multiplies inter-rack links and 100 GbE optics; the USD 900k can vanish in optics and a five-year refresh. Mitigate: model optics as a recurring line item, not capex, and standardise on one optic family to keep spares cheap.
- Vendor lock and operational fragility. A single-vendor, single-skill stack with no fallback. Mitigate: keep configuration in version control and IaC so intent survives a platform change, contract for spares and 4-hour support, and avoid features with no standards equivalent.
Explicitly accept: no in-house 24/7 eyes - that is a monitoring and escalation design problem, addressed separately.