I85PREP
⌘ K
CCNP Data Center study materials
CCNP Data Center study resources

Rack R7 Cutover: Control Plane Says Green, Bulk Transfers Say Otherwise

CCNP Data Center · Put your knowledge to work in context.

All case studies

Rack R7 Cutover: Control Plane Says Green, Bulk Transfers Say Otherwise

A 4-spine/32-leaf VXLAN-EVPN fabric with an OSPF underlay has run for 14 months. Tenants run NFS and Oracle over the fabric and need 9,000-byte jumbo frames end to end. A pair of replacement leaves in rack R7 was built from a template that was later discovered to be two years out of date. After cutover, SSH and monitoring to servers behind R7 are healthy, but file copies stall at roughly 40 KB, NFS mounts hang, and large database transactions time out. show ip ospf neighbors shows Full on every uplink, show nve peers shows all peers UP, interface counters show no errors or drops, 64-byte pings succeed in both directions, and pings with the DF bit set and a 2,000-byte payload fail.

1. State the most likely fault and the exact checks that confirm it before you change anything.

Read the worked answer

Diagnosis: an MTU defect local to the new leaves, not a routing or EVPN fault.

How the evidence reads:

  • Control plane is clean (OSPF Full, all NVE peers UP), so this is not a VNI-to-VLAN mapping problem, a missing l2-vni, or an underlay reachability problem.
  • 64-byte pings pass and DF-set 2,000-byte pings fail: forwarding works, size-dependent forwarding does not.
  • Zero errors and zero drops on counters: large frames are being discarded as giants or silently dropped, not retransmitted as a link problem would show.
  • Applications stall around 40 KB rather than returning errors: the handshake (small packets) completes, then the first full-size data burst dies.

Confirm with, in order:

  1. show interface Ethernet<x> on the new leaves' server-facing ports - is the MTU 1500 while servers are configured for 9000?
  2. show nve interface nve1 - the NVE MTU must cover tenant payload plus 50 bytes of VXLAN/IP/UDP overhead; 1500 here breaks every remote bulk flow while leaving small pings intact.
  3. show run interface ... on a new leaf versus a known-good old leaf, diffed for mtu 9216, system mtu jumbo, and any policy-map type network-qos jumbo class.

Explicitly rule out the uplinks first: if the uplink MTU had mismatched, OSPF would have stalled in EXSTART during DBD exchange rather than reaching Full, so the defect sits below the routed hop, on the access side and the NVE.

2. The change window is 90 minutes and the leaves carry production traffic. What order do you apply the fix, and how do you prove it worked without disturbing tenants?

Read the worked answer

Sequence by blast radius, keeping redundancy in place at every step:

  1. Fix one leaf of the pair, not both. Raise MTU on the server-facing interfaces and the NVE on leaf-31 only. Changing MTU on an interface is disruptive to that interface, so schedule it while vPC/port-channel peers and host teaming carry traffic on leaf-32.
  2. Prove it end to end before touching leaf-32. From a host behind leaf-31 to a host behind an old leaf in another rack, run a DF-set ping sized at 8,972 bytes (9,000 minus headers) and an NFS read of several hundred MB. A successful ping alone is not enough; a bulk transfer exercises the full path including any intermediate L2 hop still at 1500.
  3. Repeat on leaf-32, then re-run the bulk test in the reverse direction, because the return path can traverse a different leaf pair.
  4. Check the border. Tenant traffic leaving the fabric to a 1500-byte WAN will now be the new failure point; confirm a TCP MSS clamp exists on the border leaf SVIs so the 9,000-byte host MSS is rewritten downward.
  5. Close the loop on the template. The recurring risk is the two-year-old template, not this pair of switches; add an MTU assertion to the validation step of the build so the next rack cannot repeat it.

3. A colleague proposes simply lowering every server's MTU to 1500 to match the new leaves. Argue for or against, quantitatively.

Read the worked answer

Against, and the numbers make the case.

  • At 1,500-byte MTU a 64 KB NFS read is split into roughly 45 frames instead of 8 at 9,000 bytes. That is about 5.6 times the packet rate for the same throughput against the arrival rate of every leaf and spine in the path.
  • Per-packet overhead is fixed: ~50 bytes of VXLAN/IP/UDP/Ethernet encapsulation plus inter-frame gap and preamble on every frame. At 1,500 bytes that overhead is roughly 3.5% of the wire; at 9,000 it is under 1%. Losing that is permanent capacity tax on the whole fabric, not a one-time inconvenience.
  • The change also touches every server, every hypervisor profile, and every storage mount - a far larger and more irreversible change than fixing two switches.

Lowering MTU is a legitimate temporary mitigation if the fabric must stay stable overnight, and if so it should be applied at the two new leaves' tenant boundary rather than globally, announced as tactical, and reversed. The strategic fix is to bring the new leaves to the fabric standard, because the fabric standard is what the applications were sized against.