Intermittent Voice Loss on a 20G LACP Port-Channel with a Single Elephant Flow
Users on VLAN 20 report intermittent voice clipping and a 2–3% packet loss rate during business hours. The affected users are in one building connected by a two-member 20G LACP port-channel from a Catalyst 9300 stack to a core pair. The problem began after a firmware upgrade to 17.9.4a and the rollout of a new nightly backup application. Counters show Te1/1/1 at 9.6 Gbps egress and Te1/1/2 at 2.1 Gbps; both have zero input errors and zero CRC. Te1/1/1 reports 45,000 output drops per minute, almost all in the priority queue. CPU is 38%, and spanning tree has been stable for 21 days. NetFlow shows a single TCP flow carrying 7.2 Gbps from the backup server to a NAS, plus voice and ordinary user traffic. Voice is marked EF and mapped to the priority queue.
1. What is the most likely root cause, and what evidence would distinguish it from a bad cable or a duplex mismatch?
Read the worked answer
The most likely cause is LACP hashing polarization caused by a single elephant flow. A single TCP flow is hashed to one member and cannot be split across members, so the backup flow plus voice and user traffic saturates Te1/1/1 at 9.6 Gbps while Te1/1/2 carries only 2.1 Gbps. The egress port is nearly line rate, and microbursts exhaust the priority-queue buffers, producing output drops and voice clipping even though EF is mapped to the priority queue. A bad cable or duplex mismatch would show physical and Layer 2 errors: input errors, CRC, alignment, runts, late collisions, or interface resets. Here both members have zero input errors and zero CRC, and the imbalance follows the single flow, not a member. Confirm with show etherchannel load-balance, show interfaces Te1/1/1, show queueing interface Te1/1/1, and NetFlow top talkers. If the drops disappear when the backup is paused and return when it restarts, the flow is the trigger.
2. What commands or telemetry would you collect to confirm the hashing and queue behavior?
Read the worked answer
Collect the port-channel and member counters, the hashing configuration, and the queue statistics. Use show etherchannel summary to confirm both members are bundled and active, show etherchannel load-balance to see whether the hash includes Layer 4 ports, and show interfaces Te1/1/1 and Te1/1/2 to compare egress rates, output drops, and queue drops. Use show queueing interface Te1/1/1 and show platform qos queue-stats interface Te1/1/1 to confirm that the drops are in the priority queue and whether the queue limit is being hit. Use show policy-map interface Te1/1/1 to verify EF classification and priority configuration after the upgrade. NetFlow or Flexible NetFlow should identify the 7.2 Gbps flow and its 5-tuple. Also check show logging for storm control or QoS messages, and compare the behavior with the backup paused. A bad cable would show CRC and input errors on one member, which are absent here, so the telemetry should focus on egress congestion and flow placement.
3. Recommend immediate and long-term remediation.
Read the worked answer
Immediate: pause or rate-limit the backup so that the elephant flow cannot consume a member. Shape the backup at the server or apply a service policy on the backup VLAN to cap it below 7 Gbps, and move the backup window outside business hours. If the backup must run during the day, use multiple parallel TCP streams so hashing can distribute the load across both members; a single stream will still pin to one member. You can also policy-route the backup source to the less-utilized member, but that is fragile. Long-term: add a 40G or 100G uplink, or add members to the port-channel, and enable a hash that includes Layer 4 ports if the traffic mix benefits. Do not assume the priority queue protects voice when the egress port is oversubscribed; verify the priority-queue bandwidth and queue limits, and consider egress shaping on the port-channel. Validate with a repeat backup test and monitor output drops and voice quality before closing the incident.