TLS Decryption Load and HA Convergence at a Data-Center Egress
A global manufacturer runs a pair of Cisco Secure Firewall 3100 appliances in active/standby HA at its primary data-center egress. Peak throughput is 18 Gbps with 35,000 new TLS connections per second and 2.1 million concurrent sessions. To satisfy data-loss policy, the team enabled TLS 1.3 decryption with an internal resigning CA for all user traffic except finance and healthcare. Within a week, SaaS users report intermittent 30-second timeouts, and the active unit shows one Snort instance at 96% CPU while the others sit near 55%. A scheduled HA failover test produced a 40-second outage instead of the expected sub-second switch. The security team also wants to avoid decrypting server VLAN traffic and several pinned mobile applications, but the current rule decrypts everything from the /16. Management asks whether to add appliances or redesign policy, and why HA failed.
1. The team proposes adding a third appliance to spread decryption load. Evaluate that proposal and design a decryption policy that meets the stated constraints. Explain the reasoning behind each rule position.
Read the worked answer
Adding a third appliance treats a symptom. The 96% single-instance CPU indicates that decryption and Snort processing are not evenly distributed across Snort instances; adding chassis can worsen flow-hash imbalance unless the volume and type of decrypted traffic are reduced first. Build a first-match policy in this order:
- Do-not-decrypt for finance and healthcare destinations by URL/category and application, plus the known pinned mobile apps. These exceptions must precede any broad decrypt rule.
- Do-not-decrypt for server VLANs, matched by source subnet or security group.
- Decrypt only user VLANs to internet categories that require inspection.
- Default action for undecryptable traffic: alert or block according to policy, never silently bypass.
Order matters because FTD evaluates top-down; an exception placed after a broad decrypt rule will not take effect. Decrypting everything is expensive: TLS 1.3 re-signing is CPU-intensive, breaks certificate-pinned apps, and creates privacy and compliance exposure. Confirm the resigning CA chain is trusted by endpoints and verify that TLS 1.3 inspection is actually required for the selected categories. If CPU remains high after policy reduction, examine per-instance flow distribution, connection rates, and platform offload; add capacity only with evidence that policy has already been optimized.
2. Explain the 40-second HA outage and list the configuration checks and design changes you would make to achieve predictable sub-second failover.
Read the worked answer
A 40-second outage suggests that the dataplane and routing did not converge at the same time as the HA control plane. HA hellos and failover may be fast, but the upstream switch can block on spanning tree, LACP can renegotiate, and OSPF, BGP, or HSRP can take tens of seconds. Check the HA control link for latency and errors; the interface monitoring policy to see whether noncritical interfaces are triggering failover; switch ports for PortFast/edge and LACP fast; routing timers and static route tracking; whether the standby was fully synchronized; fail-open versus fail-closed inspection behavior; and platform or software defects.
Design changes: use a dedicated HA link, redundant data links in a port-channel, monitor only critical interfaces, run fast routing hellos with BFD, and provide redundant upstream paths. Test the actual failure mode, such as pulling a link versus reloading a unit, with production-like traffic. Sub-second HA is not an end-to-end guarantee; the requirement should be a bounded outage with measured convergence, not a vendor number applied blindly.
3. Which telemetry would you use to prove the redesigned policy is working, and what would you do about the pinned applications?
Read the worked answer
Use connection events with SSL policy verdicts, decryption-error reasons, per-Snort CPU, throughput, and latency, and correlate them with user reports. Track uncategorized or undecryptable traffic and certificate-pinning errors separately. Prove that server VLAN traffic and finance/health traffic are not being decrypted, and that user internet traffic still receives the intended inspection.
For pinned applications, exempt them by application ID or destination and document the residual risk. Compensating controls are endpoint protection, application allowlisting, and API-based CASB inspection rather than breaking the application. If an app is unsanctioned, block it at the application layer instead of decrypting it. Review the exemption list quarterly, because pinned apps change and stale exemptions become blind spots. Do not treat a decryption error as a reason to disable inspection globally.