Size-dependent packet loss after a 100 GbE transit link cutover
A 14-node Fabric Connect network has been stable for 18 months. A new 100 GbE transit link is added between two VSP 8600 core switches to replace a 10 GbE link. The new link is a fabric ISL carrying B-VLANs 3002 and 3003. After cutover, IS-IS adjacencies remain up on all nodes, and 64-byte pings succeed between every pair of buildings. However, large file transfers stall, VMware vMotion fails, and some HTTPS sessions hang. CFM between two edge MEPs shows 0% loss for 128-byte frames and 100% loss for 1518-byte frames. The new link is a direct fiber run with no IP routing. The old link used an MTU of 9000. The new link's interface shows MTU 1500 on one side and 9000 on the other. The team has not changed any ISID or VLAN configuration.
1. What is the most likely root cause, and why do IS-IS and small pings still work?
Read the worked answer
The most likely cause is an MTU mismatch on the new ISL: one side is 1500 and the other is 9000. IS-IS does not negotiate MTU as a precondition for adjacency, and its hellos and many control PDUs are small enough to traverse the 1500-byte side. Small ICMP also fits. The failures are size-dependent: vMotion, large file transfers, and TLS sessions with large packets need frames above 1500 bytes. When the 9000-byte side sends a large frame, the 1500-byte side cannot forward it; if Path MTU Discovery is blocked or ICMP 'fragmentation needed' is filtered, the application hangs instead of falling back.
The CFM result is the strongest evidence: 128-byte frames pass and 1518-byte frames fail at the same MEP pair. That rules out an ISID/VLAN mismatch, which would affect all frame sizes, and points to a transit-path MTU issue. Check whether the new link is actually carrying both B-VLANs; a missing B-VLAN could also be size-independent, so the size correlation matters.
2. What checks confirm the diagnosis, and what is the safe fix?
Read the worked answer
Confirm with show interfaces on both sides of the new link, show isis adjacency detail, and show isis spbm to verify the B-VLANs are active. Run DF-bit pings of increasing size (1400, 1472, 1500, 1600, 9000) across the transit path and note where they fail. Compare every ISL in the path, not just the new link, because an MTU mismatch can exist on an older link and only be exposed by the new path. Check interface drop counters for giants and overruns.
The fix is to set both sides of the new link to the intended fabric MTU, typically 9000, and verify that every fabric port in the path uses the same value. Do this in a maintenance window because changing MTU can bounce IS-IS and reconverge the fabric. After the change, re-run the large-frame DF pings and CFM at 1518 bytes, and confirm vMotion and file transfers. Do not set only one side to 9000; that preserves the mismatch in the opposite direction. If the platform supports a larger value such as 9216, use one documented fabric-wide value rather than mixing 9000 and 9216.
3. What monitoring would detect this earlier, without waiting for user reports?
Read the worked answer
Add CFM or SLA probes that test multiple frame sizes, not only small frames, between key edge pairs. A 128-byte-only probe would have shown green throughout the incident. Include a 1518-byte probe and alert on any size-specific loss. Add a pre/post-change compliance check that reads interface MTU on every ISL and fails the change if the values differ from the documented fabric MTU.
Also monitor interface giants/discards and IS-IS adjacency detail after changes, and keep a synthetic large-transfer test in the cutover checklist. Syslog alone is weak here because the mismatch may not log an adjacency event. The best control is a change gate: no new ISL is accepted until MTU, B-VLAN allowance, and large-frame CFM all pass.