Hybrid WAN for a 240-Branch Bank With a Hard 400 ms Transaction Budget
Meridian Retail Bank runs 240 branches on an MPLS VPN: 180 small branches at 20 Mbps and 60 larger branches at 100 Mbps. The teller platform, hosted in two data centers, has a hard 400 ms application timeout and is chatty, about 60 round trips per transaction. Every branch also has a 300/50 Mbps business broadband circuit used for guest Wi-Fi and Office 365. After a pilot at 14 branches moved the default route onto the broadband link, teller transactions began timing out during the 11:00-13:00 peak, and the service desk sees sessions resetting mid-transaction. MPLS utilization at those branches sits at 27%; broadband peaks at 80%. A regulatory requirement states that teller traffic must never traverse the public Internet unencrypted. The network team is six engineers who already operate the MPLS. The CIO wants a hybrid WAN design that cuts MPLS spend by 40% within 20 months without changing the teller application.
1. Diagnose the pilot failure and specify how traffic should be separated and routed at a branch.
Read the worked answer
Broadband is not the root cause; the default-route leak is. Moving the default route moved teller traffic with it. Sixty round trips against a 400 ms budget leaves about 6 ms per round trip, which is not achievable on an Internet path that is queueing at 80% utilization at peak. Even a flawless broadband circuit cannot meet that budget, so this is a design error, not a carrier quality issue.
Separate on two independent axes:
- Routing: put guest and Office 365 in their own VRF with the Internet default route. Keep the teller prefix set on the MPLS path. If you need selective breakout, leak only named SaaS prefixes or use policy-based routing matched to a defined prefix list - never a second default route with a worse metric, because a single MPLS failure silently moves teller to the Internet.
- Classification: if teller flows cannot be identified by prefix, identify them by DSCP or 5-tuple in a zone-based policy on the CE, and drop rather than back up onto the Internet.
Immediate fix for the 14 branches: restore the MPLS default, keep broadband for guest and Office 365 via policy-based routing. Then instrument before re-enabling anything - synthetic teller probes per branch with per-hop latency, jitter, and loss. Report p95 and p99, not averages: the 400 ms timeout is triggered by the tail, and an average of 90 ms will look healthy while 4% of transactions still fail.
2. How do you actually reach a 40% MPLS reduction without breaching the transaction budget?
Read the worked answer
Model the spend first: MPLS cost is roughly port size times site count, so the money is in the 180 small branches, not the 60 large ones. Realistic levers, in order of risk:
- Move Office 365 and guest traffic off MPLS entirely (already piloted, low risk).
- Downgrade small branches from 20 Mbps to 10 Mbps after confirming busy-hour teller demand, which is small but latency-sensitive, not bandwidth-hungry.
- Remove MPLS at the subset of branches where an encrypted Internet path can demonstrably meet the budget.
Decide that subset with data, not opinion: a 30-day probe per branch measuring p99 RTT and loss to both data centers over both paths at peak. A branch qualifies only if p99 stays under roughly 150 ms, leaving headroom for server time and retransmissions, loss is below 0.1%, and the site has a second carrier or 5G failover. Expect 30-50 of 240 sites to qualify, not 100.
If that misses 40%, the honest answer to the CIO is a scope conversation: either the application gets connection pooling and fewer round trips, or a different service is cut. Shrinking MPLS bandwidth below the busy-hour teller requirement and relying on QoS to protect it is the classic failure mode - QoS cannot create latency headroom that no longer exists.
3. What is required to put any regulated traffic on the Internet path, and what residual risk do you accept?
Read the worked answer
If teller traffic ever rides the Internet it must be encrypted end-to-end to a bank-controlled endpoint, with per-branch certificates and authentication - IPsec/DMVPN with unique per-site keys, or TLS terminating on a load balancer the bank operates. Never terminate at a provider CPE you do not control, and never share one pre-shared key across all branches.
Read the regulation before designing: most such rules are satisfied by encryption in transit plus segmentation plus logging, but a strict reading may require a private circuit for cardholder data. Get a written compliance ruling per traffic class before the design is locked, because that single document decides whether the hybrid carries teller at all and therefore whether the 40% target is reachable.
Residual risks to state explicitly: the branch CE and local LAN enter PCI scope, which is an audit cost as well as a technical one; the Internet path adds a failure mode that the six-person team has never operated; and the current MPLS contract may have early-termination charges that erase the savings in year one.
A defensible outcome keeps teller on MPLS where the regulator demands private transport, uses the hybrid for everything else, and renegotiates the MPLS contract rather than assuming the 40% must come from circuit count.