Clinic 7 Voice Breakup After the SD-WAN Pilot
Valley Health Partners links 14 clinics to its data center over an MPLS VPN; two pilot clinics were moved onto a new SD-WAN overlay last month. Since then clinicians at Clinic 7 report that VoIP calls to the central scheduling desk break up between 10:00 and 11:00, and that the practice-management application stalls when opening patient charts. Simple pings show 1-3 ms with no loss. A call-quality probe reports 2% loss and jitter up to 90 ms, but only when traffic is using the SD-WAN path. The clinic circuit is 100/100 Mbps and peaks at 65% utilization. At the edge, RTP is marked EF and video AF41, but the SD-WAN policy collapses every flow above 50 Mbps into one 'business' queue with a single drop policy. The clinic switch uplink counters show a rising count of late collisions and FCS errors. The carrier states its MPLS SLA covers 99.9% availability and 30 ms latency.
1. What do you investigate first, and why? Give the sequence you would follow before touching the QoS configuration.
Read the worked answer
Physical layer first, because those counters describe errors QoS cannot cause. Rising late collisions on a switch uplink mean the link is running half duplex somewhere or the cable is marginal; FCS errors mean frames arrive corrupted. Both produce loss and retransmission that look exactly like congestion to an application. A duplex mismatch also explains the time-of-day pattern: at 10:00 load crosses the point where collisions and retransmits become audible on RTP.
Sequence:
- Check and fix speed/duplex on the clinic switch uplink and the CPE handoff, replace the patch lead, then re-read counters. Errors must stop climbing before anything else is judged.
- Verify DSCP marking end to end with a capture at ingress and egress. If the SD-WAN device re-marks EF, downstream queueing is meaningless.
- Only then assess queueing and policy.
Ping latency and the carrier SLA are not evidence of health: a 1-3 ms average hides jitter, and 65% average utilisation hides microbursts. Judge with per-class statistics, not averages.
2. Is the QoS design adequate? If not, how would you redesign it?
Read the worked answer
No. One 'business' queue above 50 Mbps is the core fault: real-time voice shares a drop policy with bulk traffic, so under pressure RTP is discarded alongside a file copy-and the 50 Mbps threshold on a 100 Mbps circuit forces anything above half line rate into contention.
Redesign:
- A strict-priority or LLQ queue for EF/RTP with a bandwidth ceiling (concurrent calls × ~100 kbps, plus about 20% headroom) so voice cannot starve everything else.
- A second queue for AF41 video with guaranteed bandwidth, and a default queue for the rest.
- Preserve DSCP end to end; confirm the SD-WAN policy matches on DSCP rather than a re-marked value.
- Shape to roughly 90-95% of the 100 Mbps circuit so queueing happens on your device rather than in the carrier's policer.
Size the queues from measured busy-hour concurrency, not from round numbers. Then re-run the probe in the 10:00-11:00 window and confirm jitter for EF falls and the practice-management app stops stalling.
3. What do you ask the carrier and the SD-WAN vendor, and how do you prove the problem is fixed?
Read the worked answer
Carrier: whose measurement, and where? Ask for per-class SLA figures and their performance-management data for the Clinic 7 path over the same 10:00-11:00 window, and confirm the CE-PE handoff speed/duplex and whether they police above a committed rate. The headline 99.9% and 30 ms say nothing about jitter or loss on the voice class.
SD-WAN vendor: how the overlay treats EF, whether it re-marks, which path voice actually takes, and what the per-class drop counters show during the busy hour.
Proof: establish a before/after baseline - two weeks of per-class jitter, loss and MOS from the probe, plus call-quality records from the phone system, sampled at the same hour each day, and interface error counters on the uplink.
Success criteria agreed in advance: MOS above 4.0 and EF jitter under 30 ms during the busy hour, error counters flat, and no regression in the practice-management application. 'The carrier says the circuit is fine' is not a closure criterion.