Steering Voice Away from a Degrading Path
A contact center runs 800 voice agents across 12 sites. Voice is carried over the SD-WAN overlay to a cloud contact-center platform. Policy currently steers voice to the MPLS transport (SLA: latency < 30ms, loss < 0.5%, jitter < 10ms) with broadband as backup. Overnight, MPLS latency rises from 18ms to 45ms at 4 of 12 sites due to a carrier maintenance window. AAR has not failed voice over because the SLA probe interval is 60s and the policy is set to "best-effort" rather than "strict." Agents report one-way audio and dropped calls.
1. Diagnose why AAR did not steer voice off MPLS and what policy settings would have prevented the outage.
Read the worked answer
Two compounding failures:
Probe interval too slow (60s). Latency rose and stayed elevated; a 60s probe means up to a minute of bad voice before any decision - and if the probe samples are averaged, a sustained 45ms may take several intervals to breach the threshold. Voice MOS degrades within seconds.
"Best-effort" SLA mode. In best-effort mode, AAR prefers the best path but does not enforce the SLA - if no path meets SLA, or if the preferred path degrades, it may keep traffic on the degraded path rather than failing over or alerting. Strict mode enforces the SLA and fails over (or blocks) when the threshold is breached.
Prevention:
- Set voice policy to strict SLA enforcement.
- Reduce probe interval to 1–5s with p95/p99 evaluation, not mean.
- Add loss and jitter to the SLA, not just latency - 45ms latency with jitter may still pass loss but fail jitter.
- Configure fast failover to broadband with pre-established tunnels so switchover is sub-second.
- Alert on SLA breach so NOC sees it before agents do.
2. Broadband at these sites is 100 Mbps shared with general internet traffic. What must be true for broadband to actually carry voice acceptably during MPLS maintenance?
Read the worked answer
- Voice must be prioritized on broadband: DSCP EF classification, strict-priority queue, and a shaper reserving ~30% headroom for voice and control. 800 agents × ~100 kbps ≈ 80 Mbps if all at one site - but per site it is ~67 agents × 100 kbps ≈ 6.7 Mbps, well within 100 Mbps if queued correctly.
- Bufferbloat control: broadband without shaping will queue voice behind bulk traffic; the shaper must cap egress to ~90% of line rate.
- Jitter and loss must meet SLA: broadband's 30–50ms RTT and variable jitter may fail the voice SLA. If so, the correct behavior is to fail the call setup or alert, not to silently degrade.
- Path symmetry: ensure return path also uses broadband, or voice will see one-way audio (the exact symptom reported).
- Capacity check: confirm the site's broadband is not oversubscribed by other apps during the maintenance window; consider pausing bulk jobs.
3. After the incident, how would you validate the fix without waiting for the next carrier maintenance?
Read the worked answer
- Inject latency/loss/jitter on the MPLS tunnel (or a test VLAN) using a traffic generator or the platform's impairment tool, and confirm voice fails over within the target time (e.g., <2s).
- Measure MOS end-to-end with a voice quality probe before, during, and after failover.
- Verify one-way audio is gone by checking return-path selection and stateful firewall symmetry.
- Load the broadband with background bulk traffic during the test to confirm QoS holds voice priority.
- Review AAR logs to confirm the decision was made on the intended metric (p99 latency/jitter) and within the intended interval.
- Document the runbook so the NOC can trigger the same test before future maintenance windows.