Voice Degradation That Never Triggers Failover
A retailer operates 140 branches on Prisma SD-WAN with ION 3000 appliances. Each branch has an MPLS circuit with 20-50 Mbps committed, a broadband circuit at 300/30 Mbps, and LTE backup. Voice runs on a hosted UC platform, and the underlay SLA policy is still the built-in default: 150 ms latency, 30 ms jitter, 2% loss. Over six weeks, 22 branches raised voice tickets for clipped speech and one-way audio between 13:00 and 15:00, yet no branch failed over - the tunnels stayed on MPLS. At those branches the busy-hour load is 26-31 Mbps against a 20 Mbps committed rate. Broadband measured at the same hour shows 38 ms latency and 0.3% loss. The carrier reports the MPLS SLA as met: 42 ms round trip to the hub, 9 ms jitter, 0.6% loss. Branch QoS consists of a single business class with no shaping applied to the MPLS interface.
1. Explain why failover did not occur, and what you would change first.
Read the worked answer
Two independent reasons, and fixing only one wastes the change window.
Reason 1 - the thresholds are wrong for the application. The default 150 ms / 30 ms / 2% policy describes a tolerable data path, not a voice path. Measured MPLS is 42 ms, 9 ms jitter, 0.6% loss, which passes the default comfortably. Even a perfectly functioning probe would never have marked that path down. Voice needs per-application SLA thresholds in the region of 100 ms latency, 10-15 ms jitter and well under 1% loss, applied to the voice traffic class rather than globally.
Reason 2 - the branch is oversubscribed and unshaped. 26-31 Mbps offered against 20 Mbps committed, with no shaper on the MPLS interface, means the provider or the CE egress queue is dropping and delaying packets in the busy hour. Probe traffic either shares that congested queue and gets delayed alongside voice, or is deprioritised differently from real call media, so the probe's view and the user's experience diverge.
Change first: shape to the committed rate and give voice a strict priority class with a bandwidth reservation, then tighten per-app SLA. Shaping first is deliberate - if you invert the order you will fail voice calls onto broadband at the first congestion event while the real cause is still local queuing. Verify with per-app path quality data before and after, not with ticket volume alone.
2. Design the path-selection policy for voice and business traffic across MPLS, broadband and LTE. What would you deliberately not do?
Read the worked answer
Voice: preferred path MPLS, with broadband as an SLA-triggered alternative rather than a loss-of-signal alternative. Enable forward error correction or packet duplication on the voice class so a single lossy path can still carry a usable call - duplication costs bandwidth, so reserve it for the voice class only. Set thresholds tight enough that broadband at 38 ms / 0.3% loss wins the moment MPLS degrades.
Business and bulk traffic: least-path-cost with MPLS preferred, but allow broadband to absorb elastic traffic such as backups and OS updates so it never competes with voice for the committed MPLS rate. That is where you reclaim the 6-11 Mbps of oversubscription cheaply.
LTE: last resort only, with a hard data cap and a policy that prohibits voice on LTE except during a declared dual-circuit outage. Metered backup carrying default-routed traffic is a recurring billing incident waiting to happen.
What I would deliberately not do:
- Do not make broadband the primary for voice to avoid the MPLS conversation. 300/30 is asymmetric and the 30 Mbps uplink is uncontended today, not at scale across 140 branches.
- Do not enable immediate fail-back on a single good probe. Flapping between paths mid-call is worse than a degraded call; require a sustained recovery window.
- Do not set identical SLA thresholds for every application class - that is how you get failover storms during a routine ISP maintenance window.
3. What would you check on the data-centre side and in the operational process before declaring the issue closed?
Read the worked answer
Branch fixes are only half the path. Verify the following.
Hub and data-centre side: confirm the hub IONs terminate both MPLS and internet tunnels, that the broadband path from 22 branches does not traverse a shared internet egress that itself congests at 13:00-15:00, and that the hosted UC provider permits media from the broadband public IP ranges. A common trap is fixing the branch and discovering the hub has a single 200 Mbps internet handoff serving every failover.
Asymmetry: with 300/30 broadband, the return media path from the UC provider is fine, but any on-premises service the branch reaches over broadband may now return via MPLS. Check for stateful firewalls in the path that will drop asymmetric flows and produce exactly the one-way audio you were trying to cure.
Operational process:
- Alert on per-app path quality breaches, not only on tunnel-down events. The six-week silence is a monitoring gap, not a product gap.
- Publish the busy-hour report to the carrier; 0.6% loss against a committed rate is worth a contractual conversation even though it is nominally in SLA.
- Re-baseline the 22 branches after two full weeks, comparing MOS or equivalent call-quality scores rather than ticket counts, since users stop reporting once it improves marginally.
Close the incident only when a simulated MPLS degradation demonstrably moves voice to broadband and back without dropping an active call.