SSL Forward Proxy Broken in Three Directions
SSL forward proxy was enabled for outbound web on a PA-3220 HA pair over the weekend. Monday brings three ticket queues. (1) Finance cannot reach an expense SaaS tool: browsers show a certificate authority error, and the decryption log lists the destination category as 'unknown'. (2) A vendor API on TCP 8443 that authenticates clients with certificates now rejects every request, although the same client works when taken to a hotspot outside the firewall. (3) Between 09:00 and 11:00 show system resources reports the decrypt resource at 100%, and a video-heavy news site that used to load in under a second takes four to six seconds. The decryption policy decrypts five URL categories including 'business-and-economy', and a no-decrypt rule exempts 'health-and-medicine' above it.
1. Explain the Finance SaaS failure and give a fix that survives the next SaaS vendor change.
Read the worked answer
Why it fails: the category is unknown, which means the URL database could not classify the flow (IP-literal endpoint, CDN hostname with no useful SNI, or a lookup miss). A decryption policy built on URL categories cannot match what it cannot categorise, so the no-decrypt exemptions never apply to those endpoints, and anything that pins or strictly validates certificates breaks. Separately, confirm the forward trust CA is actually in the machine trust store - a CAUTHORITY_INVALID error is also exactly what an untrusted root produces, and the two causes look identical to the user.
Durable fix (in order of preference):
- Exempt by App-ID: a no-decrypt rule matching the SaaS application, placed above the decrypt rule, so the exemption follows the app rather than a URL category.
- Exempt by a destination EDL fed from the vendor's published ranges - resilient to CDN shuffling, with a named owner for refreshes.
- Only as a last resort, add the vendor to the SSL Decryption Exclusion list: it is per-application, needs App-ID to work, and is easy to forget.
Proof: filter the decryption log by user and reason fields, confirm the exemption matches, then re-test after the next content update - content releases change URL categories and that is when these regress.
2. The TCP 8443 API with client certificates fails. Why, and what are your options?
Read the worked answer
Why: forward proxy terminates the client's TLS session and re-originates a new one to the server, so the server's mutual-TLS challenge is answered by the firewall rather than by the client. Unless the PAN-OS release explicitly supports relaying a client certificate in forward proxy, the server sees a session with no client identity and rejects it. The hotspot test proves the path is the cause, not the vendor.
Options:
- No-decrypt rule for that destination and port (destination address/EDL or application), above the decrypt rule, with a documented compensating control (endpoint protection, restricted source group). This is the pragmatic answer.
- Exemption list entry if the endpoint is reached by a hostname the firewall can identify as an application.
- Restructure so the endpoint is not in the decryption path at all (dedicated source-based no-decrypt for the integration subnet, or an outbound proxy for that service).
Do not disable decryption globally, and do not "fix" it by turning off certificate validation for everyone. Record the exemption with an owner and an expiry date so it is revisited at the next PAN-OS upgrade, when relaying support may change.
3. The decrypt resource is saturated at peak and page loads balloon. Diagnose and rebalance.
Read the worked answer
- Diagnosis: the bottleneck is TLS handshake/crypto capacity on the dataplane, not bandwidth. 09:00-11:00 is handshake-heavy (full handshakes on new sessions, poor session reuse). Check per-DP resource monitor output, decrypt session counts, and decryption log reason codes - reasons like unsupported cipher/version tell you whether sessions are being decrypted or silently passed, and whether rejected handshakes are causing client retries that multiply the load.
- Narrow the scope: decrypt the categories that need content inspection and drop bulk categories (streaming media, software updates, backups) from the decrypt rule. The video-heavy site is the clue: bulk high-throughput categories consume handshake and crypto budget for near-zero inspection value.
- Check the decryption profile: blocking unsupported versions and ciphers converts failures into error pages and retries. During rollout, pass-through for a defined period is usually the better trade; tighten later and measure.
- Distribution: confirm no single application dominates one DP; if it does, spread egress across interfaces or use policy to split sources.
- Structural: re-size the platform on decryption throughput, not firewall throughput. If 40% of egress is to be decrypted at peak, a PA-3220-class box is likely the wrong place for it - move egress decryption to a larger chassis rather than reducing coverage.