CCNP Enterprise: VXLAN/EVPN Stretch: Two Symptoms, Two Causes

VXLAN/EVPN Stretch: Two Symptoms, Two Causes

A retailer is refreshing two data halls 6 km apart into a spine-leaf fabric: 44 ESXi hosts, 4 spines, 8 leaves, 25 GbE server ports and 100 GbE uplinks. The underlay is routed OSPF on /31 links. The team wants to stretch 60 VLANs across both halls so that vMotion and a shared NFS datastore work without re-IPing 900 VMs. They deployed VXLAN with EVPN and an anycast distributed gateway, left the underlay MTU at 1500, and enabled ARP suppression. Since go-live, pings and SSH sessions to migrated VMs drop for 4–6 seconds after each vMotion, and NFS write latency spikes to 60 ms in those windows. Small pings cross the fabric fine; 4000-byte pings through the tunnel fail silently. A capture shows the migrated VM's traffic arriving at the remote leaf but the return path blackholing until a gratuitous ARP is sent.

1. Separate the observed faults. Which symptom is MTU and which is control-plane, and what do you change for each?

Read the worked answer

Fault A - MTU. The silent failure on 4000-byte pings with small pings passing is diagnostic.

VXLAN overhead is 50 bytes: outer IP 20 + UDP 8 + VXLAN 8 + inner Ethernet 14. A 1500-byte inner frame needs 1550 on the wire. With the underlay at 1500 and DF set on the outer header (the correct configuration), the egress device cannot forward the frame, returns ICMP unreachable to the VTEP - which does not act on it - and the flow blackholes. If DF is cleared, the outer packet fragments, which many platforms drop in hardware and which loads the remote VTEP with reassembly.

Raise the underlay to at least 1600; 9000 is the right target if NFS or iSCSI shares the fabric.
Raise it everywhere: the vSwitch, the VMkernel, the physical NICs, both leaf and spine tiers, and the inter-site link. One 1500-byte hop reintroduces the silent drop for full-size frames only.
Validate with vmkping -d -s 8972 and by moving real files, not with a 64-byte ping.

Fault B - the 4–6 second post-vMotion drop is not MTU. The VM's return traffic follows the EVPN type-2 MAC route. After vMotion the MAC moves to the destination leaf, but the remote leaf's ARP-suppression cache still binds the IP to the old MAC on the old VTEP, so return traffic is tunnelled to a leaf that no longer has the MAC and is dropped until a GARP or ARP refresh updates the binding.

Confirm with show l2route evpn mac - look for the MAC-mobility extended community and a higher sequence number on the new leaf.
Check that the GARP emitted by the VM on resume is not being dropped or suppressed at the destination leaf.
Until the fabric is stable, disable ARP suppression on the vMotion subnets, or shorten the ARP sync interval, and re-enable once mobility is clean.

2. Should all 60 VLANs be stretched? Give the decision criteria and a target design.

Read the worked answer

Criteria - stretch a subnet only if an application genuinely requires same-subnet adjacency across the halls:

vMotion/DRS across sites: stretch the vMotion and management subnets only. That is two or three VLANs, not sixty.
Clustering or legacy apps with hard-coded IPs: if re-IPing is a funded project, route instead. If the budget is not there, stretch that subnet alone and put an expiry date on it.
Everything else routes. With an anycast distributed gateway the default gateway is already local on every leaf, so routing costs you nothing operationally.

Target design:

One transit/VXLAN L3VNI per hall for routed inter-VLAN traffic; per-VLAN L2VNIs only for the stretched set.
Keep the NFS datastore off the stretched set unless the end-to-end path supports 9000 bytes. NFS inside VXLAN with a 1500 underlay produces exactly the 60 ms write spikes observed - the storage traffic is the one workload that cannot tolerate the MTU corner.
Budget the blast radius honestly: a stretched subnet means one broadcast domain, one ARP cache, and one gateway-failure domain across 6 km. Six VLANs is manageable; sixty makes a single misconfigured host a two-site event.

The judgement the exam is looking for is that "we don't want to re-IP" is a project constraint, not a technical requirement, and the cost of the stretch is paid at every future outage.

3. The team proposes raising the underlay MTU to 9000 on all links. What breaks, and how do you prove it end-to-end before the next change window?

Read the worked answer

What breaks when jumbo is inconsistent: nothing at low packet sizes, which is the problem. Full-size frames are dropped or fragmented only on flows that happen to be large, so the failure looks like an application problem - slow NFS, stalled backups, one flaky VM - and the fabric pings clean.

Likely gaps in this environment:

vSwitch and VMkernel MTU left at 1500 while physical ports go to 9000. The VMkernel path for vMotion and NFS then cannot carry a 9000-byte frame.
The inter-site link or the carrier service stays at 1500 or 1600 - common, and often contractual.
Some leaf ports jumbo, some not, because the change was done with a per-interface command rather than a template.

Proof, in order:

Inventory MTU per interface and per vSwitch from the source of truth (templates/automation), not from memory.
ping with DF at 8972 across every path including the inter-site link, and vmkping -d -s 8972 from each VMkernel.
In a VM, transfer a known large file and confirm the interface counters show no drops and no fragments.
Do it in one hall first and compare against a control path.

Judgement: jumbo is a whole-path property, not a device setting. Anything that changes MTU on only part of the path is worse than leaving it at 1500 and designing the storage traffic around the limitation.