1% packet loss on one flow after a VXLAN migration

Reproduce at flow level, not with ping. The failure is a bulk-transfer failure; a successful 56-byte ping proves almost nothing. Drive traffic between the actual VMs and vary the 5-tuple and payload size deliberately.

Bisect the layers:

1. VM-to-VM on the same leaf - if it also fails, the overlay is exonerated and you are looking at host or access-layer issues.
2. Leaf-to-leaf loopbacks with a DF-bit size sweep - tests the underlay independently of VXLAN.
3. VM-to-VM across racks, varying payload size and source port - this is where the reported symptom lives.

Ranked hypotheses given the evidence (large-frame only, no error counters, no logs):

1. MTU mismatch somewhere in the path - VXLAN adds ~50 bytes, a 1500-byte hop silently drops the outer packet, and DF prevents fragmentation, so nothing is logged and no CRC appears.
2. ECMP/LAG hash polarisation - one member of the two-member LAG or one spine handling all frames for that flow, or a mismatched member config.
3. Microburst / buffer drops on a single egress queue - invisible to five-minute polling.
4. ARP/ND suppression blackhole for a silent host that stopped answering.

Evidence to gather: configured MTU on every hop (not assumed from the template), VXLAN tunnel drop counters, per-queue drop counters, and sFlow or telemetry filtered to that 5-tuple on the spines.

Prove it:

• DF-bit ping sweep between leaf loopbacks and between the two VM subnets at 1472 / 1500 / 1600 / 2000 / 8972. A cliff between 1472 and 1600 points at a 1500-byte hop; a cliff higher up points at a mis-sized VTEP interface.
• Compare the configured MTU per port on every hop by reading the config, not the template. The classic failure is one link that came up default-1500 during a rebuild, or one LAG member configured differently from its partner - both are invisible to the interface error counters.
• Check the VSX pair specifically: an ISL or a LAG member at 1500 will drop exactly the frames you are losing while passing everything small.

Durable fix:

1. Bring every fabric link and VTEP SVI to the same IP MTU value.
2. Make MTU a compliance check in the automation pipeline so it cannot regress on the next rebuild - this incident class recurs precisely because MTU lives in templates nobody diffs.
3. Re-run the DF-bit sweep as a post-change gate on the canary device per site.

If the sweep comes back clean, pivot to hashing: vary the source port on the bulk transfer. If a different 5-tuple succeeds, you have a hash-polarisation problem, not an MTU problem - fix by adjusting the hash inputs or by adding a LAG member, and confirm on both VSX peers.

The gap is resolution, not coverage. A 1% loss concentrated in bursts during bulk transfer is invisible to five-minute SNMP polling of interface counters, and clean counters are exactly what you would expect from DF-driven drops - they never count as interface errors.

Changes, in order of value per unit of effort:

1. Per-queue drop and buffer counters streamed at 10-second or faster intervals. This catches microbursts, the other silent killer in the same category.
2. Synthetic probes between racks: a scheduled DF-bit MTU probe and a sustained bulk-transfer probe. Both are cheap, both fail loudly, and the MTU probe would have flagged this on day one of the migration.
3. TCP retransmission rate as the customer-facing metric, not interface errors. Collect it from sFlow on the VTEPs or from the hosts; 1% retransmits on one flow is a business-visible symptom long before a counter moves.
4. Flow telemetry on the spines filtered per 5-tuple so hash polarisation and single-member hot spots are visible rather than averaged away.

Alerting discipline: alert on the retransmit rate and on MTU-probe failure, not on packet counts. A fabric that pages on every counter increment gets muted within a month, and then you are back to finding out from the customer.