VXLAN EVPN Stretch with a Non-VXLAN Firewall and Missing Multicast License
Your company is deploying a VXLAN EVPN fabric in two data centers 40 km apart. Each site has two Nexus 93180YC-FX spines and four leaf switches running NX-OS 10.2, managed by a central NDFC. The underlay is OSPF area 0 with /31 links and 9216-byte MTU inside each data center; the inter-site link is a 10G dark-fiber pair with 9000-byte MTU. You must stretch 300 of the 2,000 existing VLANs between sites for vMotion, but only 40 need simultaneous Layer 2 adjacency. Security requires all inter-site traffic to pass a firewall pair that does not support VXLAN. The multicast license is missing on the four secondary-site leaf switches, and procurement will take six weeks. The application team wants anycast gateway and fast failover for the stretched VLANs.
1. Choose between a multicast underlay and ingress replication for VXLAN BUM traffic. Defend the choice using the site scale, license constraint, and application requirement.
Read the worked answer
Choose ingress replication for the initial deployment. There are only eight leaf VTEPs across both sites, and only 40 VLANs need simultaneous Layer 2 adjacency, so BUM traffic is limited. The missing multicast license on four secondary-site leaf switches makes a PIM underlay impossible without waiting six weeks, and ingress replication removes that dependency entirely. Configure head-end replication on each NVE interface, listing the remote VTEPs; each ingress leaf replicates broadcast, unknown unicast, and multicast frames to the remote VTEP list. The trade-off is that the head-end sends N-1 copies, so a large BUM-heavy environment would scale poorly, but that is not this design. vMotion is primarily unicast and is unaffected by the BUM choice. Anycast gateway works independently: configure the same distributed gateway IP and MAC on every leaf that hosts the VLAN. Once the multicast license arrives, you can migrate to a multicast underlay if BUM traffic or VTEP count grows. Document the ingress replication VTEP list as part of the fabric template so that new leafs are added consistently.
2. How do you insert the non-VXLAN firewall for inter-site traffic while preserving the 40 Layer 2 adjacent VLANs?
Read the worked answer
Minimize the Layer 2 stretch rather than extending all 300 VLANs. Stretch only the 40 VLANs that require simultaneous adjacency and the vMotion VLAN. For all other inter-site traffic, use EVPN Layer 3 VNIs and route through the firewall in routed mode; the firewall becomes the next hop between VRFs or between sites, so security inspection is preserved. For the 40 stretched VLANs, the firewall cannot inspect the VXLAN-encapsulated Layer 2 frame directly. Put those VLANs in a dedicated transit VRF with strict ACLs, and accept that only that exception bypasses stateful inspection. If security insists on inspection, place the firewall in transparent bridge mode on the DCI path, but that adds a failure domain and throughput bottleneck. A better compromise is to reduce the 40 to the minimum set, route their default gateways through the firewall where possible, and document the residual exception for vMotion and any true Layer 2 requirement. Do not stretch all 300 VLANs just because vMotion exists.
3. What MTU should be configured on the DCI link and on the leaf access ports, and how does anycast gateway interact with ARP suppression?
Read the worked answer
VXLAN adds 50 bytes of overhead: outer Ethernet 14, outer IP 20, UDP 8, and VXLAN 8. If a tenant frame is 9000 bytes, the outer packet is 9050 bytes, which exceeds the 9000-byte DCI MTU and will be dropped. Inside each data center, 9216 is sufficient. For the DCI, either raise the link MTU to at least 9050 or lower the tenant MTU to 8950 so the encapsulated packet fits within 9000. Apply the lower tenant MTU consistently on leaf access ports, SVIs, and vMotion interfaces; test with DF-bit pings at the target size. Anycast gateway uses the same distributed gateway IP and MAC on every leaf, so a host can move between leafs without changing its default gateway. ARP suppression uses EVPN MAC/IP routes to answer ARP locally for known remote hosts, reducing BUM flooding. It requires consistent anycast MAC and symmetric EVPN configuration, and it does not replace end-to-end MTU validation.