SSL Decryption and IPS Saturation at the Internet Edge
A USG6650E at an enterprise internet edge carries 1.2 Gbps at peak, 800,000 concurrent sessions and 60,000 new sessions per second. After enabling IPS, antivirus and SSL decryption for outbound HTTPS, users report 8-to-10-second page loads and some TLS handshakes fail. Firewall CPU is 92 to 96 percent, with the IPS and decryption threads dominant; memory is 61 percent. Interface drops appear on the trust-side 10G interface, and the session table is at 78 percent. Packet captures show TCP retransmissions from clients but not from servers. The business wants decryption kept for 200 finance users only; currently it is applied to all 3,000 users.
1. How would you troubleshoot this in a structured way, and what would you check before changing any security profile?
Read the worked answer
Establish whether the bottleneck is inspection capacity, interface or session state, in that order. Check display cpu-usage and the per-thread or per-service view to confirm IPS and decryption dominance, then display memory-usage and session table utilization. Look at display interface brief and discard counters on the trust-side 10G interface to see whether drops are ingress buffer exhaustion or egress shaping. Check whether hardware fast forwarding or offload is active for the flows in question, because decrypted flows may be punted to the CPU and never hit the fast path.
Then isolate scope: compare latency for a finance user with decryption against a non-finance user without it, and test a known site with and without the profile. Capture on both sides and confirm where the retransmissions originate. Verify the session table is not near its limit and that no session aging anomaly is churning entries. Only after this baseline would I change profiles, because disabling IPS blindly may remove the control that is actually needed.
2. What optimization measures would you apply, and in what order, to restore acceptable latency?
Read the worked answer
First reduce unnecessary work: scope decryption to the 200 finance users with a policy that matches user group and destination categories, and bypass categories that are legally or technically inappropriate to decrypt, such as banking and healthcare. Use server-name or certificate-based exemptions rather than broad IP exemptions. Second, reduce IPS load by selecting a profile tuned to the actual traffic mix instead of the maximum signature set, and disable signatures that are irrelevant to the environment.
Third, check that traffic takes the fast path where possible and that the decryption policy does not force all HTTPS through a single inspection path. Consider offloading decryption to a dedicated service card or a separate inspection cluster if the platform supports it. Fourth, address the trust-side drops: confirm whether they are caused by CPU punt or by burst absorption, and adjust interface queues or upstream shaping accordingly. Re-measure latency, CPU and drop counters after each step so you know which change produced the gain.
3. The business wants decryption for all 3,000 users again within six months. What capacity and design decision would you present?
Read the worked answer
Present the measured cost per decrypted flow and the headroom: at 92 to 96 percent CPU with decryption for 3,000 users, the device has no failure margin, and a single link or card failure would push it into overload. Calculate the peak concurrent decrypted sessions and throughput for the full user base, add at least 30 to 40 percent headroom for bursts and failover, and compare that with the platform's rated IPS and decryption throughput, not its firewall throughput. Those numbers are different and the marketing figure is usually the firewall-only case.
The design options are: scale out with a second inspection node and distribute users, add dedicated service hardware if supported, or tier decryption by risk so only high-risk destinations and user groups are decrypted. Recommend the tiered approach plus measured capacity growth, with a pilot that validates latency at peak before expanding. Do not commit to full decryption on the current chassis without a load test that includes TLS 1.3, because the handshake cost, not just throughput, is often what saturates the CPU.