SaaS App Alternately Allowed and Blocked After ECMP Change
Harborline Retail runs a PA-3410 active/passive pair at headquarters, serving 3,200 employees with a 1 Gbps fiber internet circuit and a 500 Mbps fixed-wireless backup. Guest traffic is steered to the backup circuit by a policy-based forwarding rule. After a core switch refresh, the switch pair load-balances return traffic with ECMP across links that land on both firewall peers. Within a week, 11-14 percent of sessions are logged with the application ssl or insufficient data, and the same SaaS CRM is alternately allowed and blocked by a decryption policy that has no exclusions. The CRM's companion desktop client, which pins the vendor certificate, now fails with TLS errors for about 200 users. The network team says routing is correct because pings succeed, and proposes disabling SSL forward proxy for all users to make the tickets stop.
1. Explain what `ssl` and `insufficient data` actually mean in the session table, and give the minimum design change you would make. What evidence would you collect before touching policy?
Read the worked answer
ssl and insufficient data are admissions of missing evidence, not application identifications. App-ID needs the handshake in both directions, client hello plus server hello and certificate, to name an application; with only the client-side packet it stops at ssl, and with almost nothing it reports insufficient data.
Evidence first:
show session all filter application ssland look for many entries with tiny packet counts in one direction only.show session id <id>and compare source and destination byte counters; a near-zero return counter confirms one-way visibility.show counter global filter delta yes | match flowfor out-of-order, asymmetric or discard counters; capture on both peers for a single user.
Pings succeed because ICMP is stateless and either path returns the reply. TCP is not forgiving: with ECMP, SYN and ACK traverse different peers, so the active device never sees the server response. In active/passive, traffic that lands on the passive peer is simply dropped.
The minimum fix is to restore symmetry, not to weaken inspection: remove the ECMP path across both peers, or use policy-based forwarding so the active firewall is the only next hop for both directions, and disable ECMP or constrain hashing to keep flows on one link. Disabling decryption hides the symptom and converts a classification problem into a blind spot.
2. The pinned desktop client cannot tolerate forward proxy. Which decryption exclusion would you write, and what residual risk do you accept?
Read the worked answer
Decryption policy cannot match on App-ID, so a rule keyed to crm-desktop is not available. That leaves three practical scopes:
- Source-based (all traffic from a user group): too broad, blinds you to every other destination those 200 users reach.
- URL-category based using the vendor's actual domains in a custom URL category: narrow, reviewable, and easy to document. This is the choice, provided the vendor publishes its domain set and you can pin the destination addresses as a second condition.
- Blanket no-decrypt for a whole zone: never, for a business application.
Residual risk is real and must be stated: this flow is no longer content-inspected, so WildFire and file blocking do not apply, and any compromise inside the pinned TLS session is invisible. Compensate by restricting the no-decrypt rule to the vendor domains and addresses, keeping App-ID visible from the certificate and SNI, applying an egress allow-list so those users cannot reach arbitrary destinations, and asking the vendor or MDM to trust an enterprise CA if a future client version allows it. Certificate pinning cannot be intercepted without breaking the application; the correct answer is a documented, narrow exclusion with compensating controls, not a global proxy shutdown.
3. The vendor engineer suggests converting the pair to active/active to spread the asymmetric flows. Evaluate that proposal.
Read the worked answer
Active/active does not repair missing return traffic. It changes which device owns a session, not whether both directions arrive. With ECMP hashing per flow, a return packet can still hash to a peer that does not own the session; PAN-OS drops non-owner traffic, and in active/active the problem shifts from black holes to resets and, if session setup or floating addresses are configured carelessly, to duplicate or out-of-order packets.
Active/active is appropriate when you need to scale throughput across peers and you can guarantee that each flow is hashed symmetrically, for example with a single upstream router using stable Layer 3 hashing and floating IP addresses with ARP suppression. In this environment the core switch pair is the source of the asymmetry, so active/active would add HA complexity without fixing the root cause.
Decision: keep active/passive, fix forwarding so both directions traverse the active peer, verify with session counters, then revisit HA mode only as a throughput exercise with a documented symmetric-hashing design.