Active-active data centers with EVPN-VXLAN and a stretched Oracle RAC VLAN
FinBank is building a new active-active pair of data centers, 36 racks each, with 25 Gbps servers and 100 Gbps spine uplinks. Each site will have four leaf pairs and two border leaf pairs. The bank needs Layer 2 extension for a legacy Oracle RAC cluster between sites, anycast gateways for tenant VRFs, and active-active firewalls in each DC. Server teams insist on keeping some VLANs stretched across sites and want ARP suppression. The underlay is eBGP with a single AS per site; the overlay will be EVPN-VXLAN. The network team is debating edge-routed bridging versus centrally routed bridging and whether to use EVPN multihoming at the leaf. MTU must support VXLAN encapsulation, and the WAN between sites is 100 Gbps with 2 ms latency.
1. Recommend ERB or CRB for this design and explain the implications for anycast gateway, firewall insertion, and failure domains.
Read the worked answer
Recommend edge-routed bridging (ERB) for the tenant VRFs. With ERB, each leaf runs the distributed anycast gateway and does local L3 routing for its attached hosts. Traffic between hosts on the same leaf stays local; traffic between leaves uses EVPN type-5 routes over the VXLAN overlay. That reduces hairpinning to the border and keeps ARP/ND domains small.
CRB centralizes routing at the border leafs. It simplifies firewall insertion because all inter-VRF and north-south traffic passes through a small number of border nodes, but it creates a larger failure domain and can force east-west traffic through the border even when both endpoints are in the same rack row. For an active-active pair of data centers with many tenants, CRB becomes a scaling and blast-radius problem.
Firewall insertion with ERB: use a dedicated service VRF and route-leak through the active-active firewall cluster at the border. The firewall pair can be dual-homed with ESI-LAG to the border leafs, and policy is applied between tenant VRFs and the service VRF. Keep the firewall state synchronized across the pair and design asymmetric-flow handling explicitly.
Failure domains: ERB confines a leaf failure to its attached hosts and any local anycast gateway state; EVPN multihoming gives dual-homed servers an alternate leaf. Do not stretch the firewall cluster across sites unless the vendor supports active-active global state; otherwise keep site-local firewall pairs and route between sites through the WAN.
2. Design the EVPN-VXLAN overlay for the stretched Oracle RAC VLAN and ARP suppression. Address multihoming, MTU, and underlay routing.
Read the worked answer
Put the Oracle RAC VLAN in its own MAC-VRF and stretch it across the two sites with EVPN type-2 MAC/IP routes. Use a site-local anycast gateway for the subnet so hosts can route locally, but keep the L2 domain as small as the RAC requirement allows. Enable ARP suppression on the leafs so ARP requests are answered from the EVPN proxy table instead of flooded across the WAN. Verify that the RAC cluster’s heartbeat and multicast behavior survive suppression; if it uses multicast, enable PIM-SM in the underlay or use EVPN multicast with assisted replication, and test before production.
Multihoming: dual-home RAC servers to two leafs using ESI-LAG and EVPN all-active multihoming. That provides active-active links and avoids STP. If the servers cannot LAG, use single-homed connections to different leafs and rely on EVPN for MAC mobility, but document the failure behavior.
MTU: set the underlay to at least 9216 bytes end-to-end. VXLAN adds roughly 50 bytes of overhead, so a 9000-byte payload needs jumbo frames on the underlay and the WAN. Validate the 100 Gbps WAN path, including any provider equipment, because a single MTU mismatch will break large frames while small pings still work.
Underlay routing: use eBGP between leafs and spines with unique peer addresses and a single AS per site. For the EVPN overlay, use iBGP with the spines as route reflectors, or eBGP with allowas-in if the design requires it. Keep underlay and overlay failure detection separate so a BGP flap does not immediately tear down every EVPN session.