Daily 10:00 POS Timeouts Behind a 45% Utilization Reading
Tarnfield Retail runs a 200 Mbps MPLS hub-and-spoke from a data center CE to 60 stores, with an IPsec internet backup that is not used for production traffic. Store POS transactions hit a SQL application in the DC and have timed out daily between 10:00 and 11:00 for three weeks. The provider reports a clean SLA with no drops on the demarcation, and the hub CE port shows 45% peak utilization on the NMS, which polls every five minutes. The hub CE QoS policy shapes to 200 Mbps with classes: voice EF 10%, business-critical AF31 30%, default 40%, scavenger 5%, leaving the remainder unassigned. NetFlow at 1:100 sampling shows TCP retransmits of 2.8% between stores and the SQL servers in the busy hour, and interface output drops on the hub CE subinterface climb steadily during the same window. Voice quality stays good all day. One store's 120 Mbps circuit is connected to a switchport left at 100 Mbps/half duplex after a replacement. The DC firewall and load balancer sit before the SQL servers, and the load balancer idle timeout is 300 seconds.
1. How can the link show 45% utilization while drops climb, and where would you measure to find out?
Read the worked answer
A five-minute average is the wrong instrument. Forty-five percent of 200 Mbps averaged over 300 seconds is compatible with one-second bursts far above 200 Mbps, and the shaper or a queue tail-drops those bursts while the average stays unremarkable. Sixty stores synchronising at 10:00 (shift start, price updates, backup jobs) is a classic burst pattern. Sampling compounds the problem: 1:100 NetFlow can miss short spikes and makes the 2.8% retransmit figure approximate, though it is still far above the sub-0.1% expected on a clean path and well above the roughly 0.5-1% tolerable on a WAN.
Measure here instead:
- Per-class queue and shaper drop counters on the hub CE (
show policy-map interface), read at 10-second intervals or via streaming telemetry rather than SNMP five-minute polls. - Interface output drops/discards and buffer utilization on the subinterface, aligned to the POS timeout timestamps.
- IP SLA or BQM probes (jitter, latency, loss) toward each store's POS subnet, so the affected path is identified per store rather than in aggregate.
- NetFlow at 1:1 or 1:10 for the busy hour on a small store subset to get a trustworthy retransmit and top-talker picture.
Drops are the ground truth: if output drops rise only during the window, offered load is exceeding the shaping rate somewhere, regardless of what the average says.
2. How would you separate congestion at the hub, the half-duplex store link, and the load balancer timeout as causes?
Read the worked answer
Run the three hypotheses against distinguishing evidence rather than assuming the busiest-looking one is at fault.
Hub congestion. If all 60 stores slow at 10:00, the shared CE is implicated. Look for per-class drops, especially in AF31: if POS traffic is marked default at the store edge and competes with bulk data, it is dropped while voice in EF stays perfect - which matches the observation that voice is fine. Check DSCP marking and trust boundaries at each store edge.
Half-duplex store link. If one store accounts for most tickets, check show interface for late collisions, runts, FCS errors and duplex mismatch on that 100/half port; a 120 Mbps circuit on a 100 Mbps half-duplex port will also be capped and error-prone. This is a local fix and should be corrected regardless, but it cannot explain 60 stores timing out.
Load balancer idle timeout. A 300-second idle timeout on pooled SQL connections produces connection resets and reconnects, not interface drops. Signature: LB and SQL logs show connection churn and resets at the same time daily, the SQL server sees connection counts spike, and the CE shows no corresponding drop increase. If the CE drops and the LB logs both look bad, the drops are the primary cause and the reconnect storms are a symptom of slow transactions.
Correlate per-store latency, per-class drops and LB connection events on one timeline; the first metric to deviate is the cause, the others follow.
3. What change would you make, and how would you prove it worked without damaging voice quality?
Read the worked answer
Change.
- Fix the store port: 1 Gbps full duplex or an appropriately sized circuit, with error counters checked afterwards.
- Correct classification at the store edge so POS is marked AF31 and trusted at the CE; if POS is currently landing in default, it is competing with bulk traffic for a 40% class and losing.
- Rebalance the policy map using measured busy-hour offered rates. If AF31 offered load peaks near 45 Mbps, 30% (60 Mbps) is nominally enough, but only if the class is not shared with other traffic - check what else is marked AF31 and split it. Keep EF untouched; voice is already healthy and is the one thing not to disturb.
- Enable WRED or ECN on the default and scavenger classes so TCP backs off before tail drop, and confirm the shaper still matches the contracted rate.
- If connection churn is confirmed, raise the load balancer idle timeout above the SQL connection lifetime rather than guessing.
Validation and safety. Apply to a canary store group first, then in batches, in a low-traffic window. Prove success with: retransmits below 1% (ideally under 0.5%), no AF31 drops during 10:00-11:00, POS p95 transaction latency back within baseline, IP SLA stable, and EF jitter/latency unchanged. Keep a rollback configuration for the policy map, monitor for 48 hours including two busy hours, and record the pre-change and post-change counters in the change record.