Intermittent HTTPS Stalls on a Decryption Gateway
Users at a 3,000-seat site report intermittent 30 to 45 second stalls when accessing a SaaS CRM over HTTPS. The firewall is a PA-5220 running PAN-OS 11.1 with forward proxy decryption for the CRM category and no decryption for financial sites. The stalls occur only for users behind a specific aggregate interface and only during 10:00-11:00 and 14:00-15:00. The firewall shows 82% session table utilization, 1.2 Gbps throughput, and 14,000 new sessions per second at peak. Global counters show tcp_out_of_window and tcp_reassembly_error increasing; threat logs show no drops. A client-side packet capture shows retransmissions and duplicate ACKs; a server-side capture is not available. The SaaS provider requires TLS 1.2 and supports TLS 1.3.
1. Outline a structured troubleshooting plan using PAN-OS tools. Which counters, logs, and captures would you prioritize?
Read the worked answer
I would start at the aggregate interface, not at the application. Check member link status, LACP state, interface errors, and per-member traffic distribution; out-of-order packets from an imbalanced or flapping LAG can produce retransmissions and duplicate ACKs on the client side. Then check session table utilization and session detail for the affected flows, including whether the firewall is doing proxy decryption, what TLS version is negotiated, and whether the sessions are hitting age-out or resource limits.
Prioritize global counters for tcp_out_of_window, tcp_reassembly_error, session allocation failures, and decryption failures. Review decryption logs for certificate or handshake errors, and traffic logs for the CRM application and source subnet. Run a packet capture on the firewall ingress and egress with a filter for the CRM IP and port 443, and compare client-side retransmissions with firewall-side forwarding. If possible, capture on a mirror port at the server side or use the SaaS provider's logs to confirm whether packets leave the firewall.
2. What are the most likely causes of the stalls, and how would you isolate decryption from network path issues?
Read the worked answer
The most likely causes are MTU or MSS problems on the decrypted path, asymmetric or out-of-order forwarding across the aggregate interface, session table pressure, or decryption dataplane overload during peak hours. Proxy decryption terminates and re-originates TLS, so MTU and MSS handling changes; if the client or server path has a lower MTU, large TLS records can stall while small transactions succeed. The tcp_out_of_window and reassembly counters point more toward packet reordering or loss than toward a pure policy block.
To isolate decryption, create a test policy for a small user group that bypasses decryption to the CRM while keeping the same routing and aggregate interface. If the stalls disappear, focus on TLS version, cipher, certificate, and MSS behavior in the decryption profile. If they remain, focus on LAG hashing, interface errors, and session table. Compare TLS 1.2 and TLS 1.3 flows separately, because the provider supports both and the decryption profile may treat them differently.
3. Recommend remediation and validation steps without disrupting the 3,000 users.
Read the worked answer
I would stage changes after hours and validate with a canary group first. Remediation candidates: clamp MSS on the decryption path, adjust the aggregate interface hash or remove a marginal member link, raise session table headroom if the platform supports it, and tune the decryption profile for the CRM category. If session table utilization is the trigger, reduce unnecessary session aging or move low-risk traffic off the firewall rather than simply raising limits.
Validation should include a before-and-after capture for the same CRM transaction, counter checks for tcp_out_of_window and reassembly errors, and synthetic HTTPS transactions from the affected subnet at peak time. I would also monitor dataplane CPU and decryption status during the 10:00 and 14:00 peaks. Do not declare success from a single user test; require a full business cycle and compare stall reports against the change window.