ERP Slowdown in the AF31 Queue Every Afternoon
At a 300-person branch, ERP invoice posting has degraded from about 4 seconds to 40–60 seconds every weekday between 13:45 and 15:30. The branch connects to HQ over a 50 Mbps MPLS circuit with four QoS classes: EF 30% for voice, AF31 25% for ERP, AF11 20% for backup, and default 25%. The ERP client opens many short TCP/1521 sessions, and responses average 2 MB. NMS shows average circuit utilization of 62%, zero input errors, and output drops rising by roughly 30,000 per minute - all in the AF31 queue - only during the incident window. Voice MOS stays at 4.2 and there are no call-quality complaints. The backup team moved the nightly database replication to 13:30 to avoid a conflict with an overnight job; it is marked AF11 at the source. OSPF and BGP are stable, and the provider's SLA reports 0.1% loss measured at ingress to its network. No QoS policy has changed in two years.
1. Build a testable hypothesis for the AF31 drops. What evidence supports it and what contradicts it?
Read the worked answer
Hypothesis. The AF31 queue is being overrun by bursty ERP responses, and possibly by backup traffic that is not actually landing in AF11. Tail drop in a 12.5 Mbps queue (25% of 50 Mbps) makes TCP collapse: a 2 MB response serialized at 12.5 Mbps already takes about 1.3 seconds, and ten concurrent sessions queue roughly 13 seconds of data. When the queue overflows, the retransmissions and window reductions turn 4 seconds into 40–60.
Supports.
- Drops are confined to AF31 and to the exact window in which the backup now runs - a timing coincidence that is too strong to ignore.
- EF stays clean and MOS is 4.2, so the circuit and the provider are not generally congested. Average utilization of 62% is consistent with bursts that briefly exceed the class share, not with steady saturation.
- Zero input errors rules out the local access circuit and the CPE port.
Contradicts or complicates.
- If backup were truly AF11, AF11 should also show drops; only ERP dropping suggests either the backup is mis-marked or is being copied into AF31 by an appliance or tunnel that rewrites DSCP.
- The provider's 0.1% ingress loss is measured before the branch egress shaper, so it says nothing about local output drops.
- A pure ERP burst hypothesis would predict the slowdown whenever invoice posting peaks, not only after 13:45.
Next evidence. Read the shaper and queue configuration, capture DSCP at the CE ingress during the window, and pull NetFlow per DSCP to see whether the backup is arriving as AF31.
2. Recommend remediation with concrete numbers and a sequencing plan.
Read the worked answer
Sequence matters: fix classification before buying bandwidth.
- Marking. Capture at the CE during 13:45–15:30. If backup arrives as AF31, correct the marking at the source or re-mark at ingress with a policy matching the backup server and port. Apply a policer to backup at 6–8 Mbps so it cannot consume the class it is not assigned to.
- Schedule. Move the replication to 19:00 or later, or at minimum stagger it so the first full copy does not overlap invoice posting. This is free and removes the trigger while classification is fixed.
- Rebalance classes. Voice at 30% of a 50 Mbps circuit is 15 Mbps for what is realistically 30 G.711 calls at about 2.4 Mbps; 20% is ample. Set EF 20%, AF31 35–40% (17.5–20 Mbps), AF11 15%, default 25%. Enable WRED on AF31 and AF11 so TCP flows back off gradually instead of all dropping at once.
- Check TCP. The bandwidth-delay product at 50 Mbps and 60 ms RTT is about 375 KB. If the ERP server's window is 64 KB, a single flow cannot exceed roughly 8.5 Mbps regardless of queue depth - which is why many parallel short sessions help and why the queue still overflows.
- Capacity. Only if steps 1–4 leave p95 posting time above target should you add a second circuit or upgrade the MPLS.
Define acceptance criteria before changing anything: ERP p95 under 10 seconds between 13:45 and 15:30, zero AF31 output drops, MOS unchanged.
3. How would you rule out the server, LAN and application before accepting a network root cause?
Read the worked answer
Network teams routinely over-diagnose themselves. Before touching QoS, establish that the delay is on the WAN path.
Server and application. Pull the ERP application logs for the same invoices: server-side processing time per request, database lock waits, and batch jobs scheduled in the 13:30–15:30 window. If server time is 2 seconds and client time is 45 seconds, the network is implicated; if server time is 40 seconds, it is not. Check whether the slowdown follows the user or the location - a control test from another branch at 14:00 is the cheapest discriminator.
LAN. Check the access-switch port counters for the ERP clients: output drops, duplex mismatch, and utilization on the uplink to the CE. A 1 Gbps LAN carrying 2 MB responses will not be the bottleneck, but a saturated 100 Mbps closet uplink would be.
WAN evidence.
- Packet capture at the branch LAN and at the CE: compare request and response timestamps to see where the seconds are lost, and look for retransmissions and zero-window events.
- A controlled iperf3 test at 14:00 and again at 19:00, single flow and eight flows, with the same DSCP marking.
- NetFlow top talkers by DSCP for the window.
Provider. Ask for per-class egress statistics from the provider, not just ingress loss - the 0.1% ingress figure does not cover the branch's own shaper.
Only when the gap between client and server timestamps lines up with AF31 queue drops should the QoS change proceed.