Voice Pinned to a Brownout MPLS Circuit in Catalyst SD-WAN
A Catalyst SD-WAN fabric has 45 branches, each with an MPLS TLOC (color mpls) and a broadband TLOC (color biz-internet) on a single cEdge. Voice is marked DSCP EF and must stay under 150 ms latency and 1 percent loss. An application-aware routing policy with two SLA classes is bound to a site list containing all branches. During a four-minute MPLS brownout (latency about 300 ms, 3 percent loss, no total loss), calls stayed pinned to MPLS for almost the entire event and only moved to broadband near the end. BFD is left at defaults, there is no centralized control policy, a single vSmart is deployed, and the site list and app-list have been verified as correct.
1. The path should have moved within seconds. Reconstruct the timeline that produced minutes, and state what you would change in the policy.
Read the worked answer
Two detection mechanisms exist, and neither one fires on a brownout by default:
- BFD declares a TLOC down only when probes stop arriving altogether. At the default 1 s hello and multiplier 7 that is roughly 7 s of total loss. A link that is delivering packets slowly and lossily keeps BFD up, so no TLOC withdraw and no control-plane failover.
- Application-aware routing is what compares latency and loss, but it does so against per-tunnel statistics at the app-route poll interval, whose default is 120 s. The first poll after degradation starts measures, the next one acts.
So the observed behaviour - pinned for nearly the whole four minutes, moved near the end - is consistent with an evaluation cadence of roughly two minutes, not with a routing bug. The site list and app-list being correct is why it moved at all.
Changes:
policy
app-route-policy AR-VOICE
vpn-list VPN-1
sequence 10
match app-list VOICE
action sla-class VOICE-SLA strict
action backup-sla-preferred-color biz-internet
action sla-class VOICE-SLA fallback best-path
plus app-route poll-interval 10 (the option supports values well below the 120 s default; validate against your release notes) and, if the platform allows, tighter BFD timers on both TLOC interfaces.
Two details that matter more than the interval: strict stops the tunnel flipping into an SLA class it cannot meet, and an explicit fallback decides what happens when no TLOC meets the SLA - without it, the behaviour is to stay on the least-bad path, which in this incident was the brownout MPLS link.
2. After the poll interval was fixed, a specific branch still sends voice over MPLS in one direction only: outbound to the data centre goes over broadband, return traffic comes back over MPLS. Where do you look?
Read the worked answer
Application-aware routing is applied to traffic as it is forwarded into a tunnel at the device making the decision. It does not describe the end-to-end path, and it does not touch the reverse direction. So a one-directional fix is the expected signature of a policy that is only effective at one end of the flow.
Check, in order:
- The return-side device. For branch-to-DC voice, the return path is chosen by the DC-side router (or the vSmart-centralized data policy instantiated on it) using its own app-route policy and its own view of the SLA. If the DC has no app-route policy, or its SLA class is bound to a different site list that does not include the DC site, it will happily keep using MPLS.
- Symmetric policy placement. Branch-to-branch voice needs the policy at both branches; a site list that only enumerates the 45 branches and omits the DC produces exactly this asymmetry.
- Match criteria on the return flow. The app-list must match traffic originating from the CUCM subnet, not the branch subnet. If the list matches a 5-tuple defined for 172.16/12 branch space, return packets sourced from 10.x DC space never match and fall through to normal routing.
A useful sanity check: show sdwan app-route stats on both ends. If the DC shows no app-route entries for the voice flow, the problem is match or policy placement, not the SLA measurement.
3. A colleague proposes adding a centralized control policy that permits only the MPLS TLOC for the voice VPN, to guarantee voice quality. Evaluate the proposal.
Read the worked answer
It will make the situation worse, and it will do so silently.
Control policy is evaluated first and shapes VPN topology - which TLOCs are even eligible for the VPN. Data policy, including application-aware routing, then chooses among the eligible TLOCs. If control policy restricts the voice VPN to MPLS, the broadband TLOC is removed from that VPN's topology before the app-route policy ever looks at it, so the app-route policy has no alternative and cannot steer, no matter how the SLA classes are configured. You would still see SLA violations logged, with no failover.
The legitimate use of control policy here is the reverse: guarantee that both TLOCs are present in the voice VPN (a partial mesh or hub-and-spoke topology that includes both colors), and use data policy for steering. If you genuinely want voice to prefer MPLS under normal conditions, express that as SLA preference or a preferred color in the data policy rather than removing the backup path from the topology.
If a control policy restriction is unavoidable for compliance reasons, the mitigation is to make the intended behaviour explicit: keep a second VPN or a hub path with broadband reachable, and verify with show sdwan policy from-vsmart on the cEdge that the broadband TLOC is still in the VPN's TLOC list before declaring the change safe.