POS Timeouts Behind a Healthy SD-WAN Dashboard
A 220-store retail chain migrated to an SD-WAN overlay six months ago. Each store has a 300 Mbps fiber primary, a 100 Mbps LTE backup, and a 1 Gbps handoff to the provider. POS transactions are HTTPS to a central gateway; nightly backups run from 01:00 to 04:00 to a cloud bucket. Since the migration, cashiers report 3–8 second POS timeouts at 12:00 and 18:00, but only at about 40 stores. The provider SLA dashboard shows 99.98 percent availability and 35 percent average utilization. SNMP polls every 5 minutes and shows no interface errors or CPU spikes. The SD-WAN controller reports path quality as good because it probes every 10 seconds with small packets. The service desk has 12 tickets and no packet captures. Store switches are 1 Gbps, and POS terminals negotiate 100 Mbps.
1. Why can the provider and SD-WAN dashboards look healthy while cashiers see multi-second POS timeouts?
Read the worked answer
The dashboards are measuring averages and synthetic probes, not the actual POS traffic pattern. A 5-minute SNMP poll reports average utilization. A 5-second burst at 420 Mbps adds about 2.1 Gb over the interval; averaged over 300 seconds, that is roughly 7 Mbps, which is invisible. The 300 Mbps shaper can still overflow during those 5 seconds and build a queue, causing multi-second delay or drops.
The SD-WAN controller probes every 10 seconds with small packets. Those probes do not fill the shaper queue, so they report good latency and loss even while a POS HTTPS burst is stuck behind a full queue. The 40 affected stores are likely the higher-volume locations with the sharpest lunch and dinner transaction bursts. The 100 Mbps POS negotiation and 1 Gbps store switch are not the bottleneck; the 300 Mbps WAN shaper is.
The missing evidence is sub-minute egress queue and drop data, not more reachability testing. The team also has no packet captures at the affected stores, so it cannot correlate a POS timeout with a specific queue drop or path change.
2. Design a measurement plan that will isolate the cause across the 40 affected stores without capturing packets at all 220 sites.
Read the worked answer
Use a comparison design: 40 affected stores plus 10 matched control stores of similar size and transaction volume.
Collect at each selected store:
- 1-minute or streaming telemetry from the WAN router for interface utilization, egress queue drops, buffer occupancy, and per-class bytes. If the router supports only 5-minute SNMP, poll the drop counters directly and compute deltas.
- IPFIX or NetFlow with 1:1000 sampling to identify top talkers during 11:45–12:15 and 17:45–18:15. Look for backup, video, guest Wi-Fi, or software updates competing with POS.
- A synthetic HTTPS probe every 2–5 seconds to the payment gateway, recording connect time and time to first byte.
- Packet captures on five affected stores and two controls, mirrored at the WAN port, during the lunch and dinner windows. Capture on the POS VLAN as well if the switch supports it.
Define thresholds in advance: egress drops greater than 0.01 percent, 95th percentile utilization above 70 percent of 300 Mbps for 5 minutes, synthetic connect time above 1 second, or RTT above 150 ms. Correlate the first failing synthetic probe with queue counters and POS terminal logs. Also check whether LTE failover activated; 80 ms LTE RTT would worsen POS timeouts but should appear as a path change.
3. Preliminary data show 5-second bursts of 420 Mbps at 12:00, egress drops on the 300 Mbps shaper, 20 ms fiber RTT, and no LTE failover. What do you recommend, and how would you validate it?
Read the worked answer
The burst exceeds the shaper by 40 percent for 5 seconds, so the queue builds and POS transactions wait or drop. The fiber path is healthy at 20 ms, so the problem is local oversubscription and queueing, not the carrier.
Recommend a two-part fix:
- Apply per-store QoS before the WAN: place POS HTTPS in a priority or guaranteed class with at least 30 percent of the shaper, cap backup and guest traffic, and enable egress shaping on the LAN side so the WAN queue does not absorb the burst.
- For the 40 highest-volume stores, raise the shaper to 500 Mbps if the QoS pilot still shows drops. A 420 Mbps burst against 500 Mbps leaves 16 percent headroom, which is thin but better than 300 Mbps; confirm the carrier can support it and that the 1 Gbps handoff is not the limit.
Validate on five stores for two weeks: 1-second telemetry, synthetic POS probes every 5 seconds, and cashier-reported timeouts. Success criteria are zero POS timeouts, egress drops below 0.01 percent, 95th percentile utilization below 70 percent, and synthetic connect time under 1 second during the lunch and dinner windows. If the pilot passes, roll out in waves of 10 stores and keep the control group to prove the change, not the season, caused the improvement.