Four Million Sessions and the Request for a Bigger Box
Every weekday at 09:00, users at a 6,000-staff site report slow browsing and scattered HTTPS failures. The USG6650E shows CPU at 87%, the session table at 3.92 million of a 4 million maximum, and NAT port-allocation failure logs appearing every few seconds. Top session consumers are VLAN 10.10.5.0/24 (a lab segment covered only by the general trust→untrust permit), with about 900,000 sessions to a single external address 198.51.100.77 on TCP 443, plus one host 10.10.5.88 generating SYNs to 10.10.0.0/16 and 172.16.0.0/16 on TCP 445. TCP session aging is at default and long-connection is not enabled. The customer has asked whether to raise the session limit or buy a larger model, and wants an answer by Friday.
1. Which observations separate 'genuine capacity shortfall' from 'enforcement problem', and what would you run to settle it?
Read the worked answer
Separate the two by looking at shape, not size. Genuine load follows the user population and grows smoothly through the morning. What is described does not: 900,000 sessions from one lab VLAN to a single destination address is pathological - a browser population does not hold 900k concurrent flows to one IP - and a single host sending SYNs on 445 across two private /16s is scanner or worm behaviour, not user traffic.
Commands, in this order:
- Session table utilisation and top talkers:
display firewall session tablefiltered by source and destination, plus the per-source top-N view, to attribute sessions to hosts rather than to the VLAN. display firewall session table verboseon a sample of the 198.51.100.77 flows to read state and remaining lifetime - thousands of half-open or long-lived idle entries mean aging, not bandwidth.display firewall statistic systemand CPU breakdown; if the data-plane/forwarding component dominates, sessions and lookups are the driver, and policy or aging fixes will help more than hardware.- Check NAT port utilisation per pool address - port-allocation failures mean the pool, not the session table, is the binding constraint.
- Confirm with a packet capture on the lab VLAN to see whether the SYNs are answered (scan) or half-open (spoofed/random).
If after enforcement the session count falls back to a normal baseline and CPU follows, it was enforcement. If the baseline is still near the ceiling at normal working hours, that is the case for capacity.
2. Give your recommendation for Friday, including the specific controls and why 'raise the session limit' is the wrong first move.
Read the worked answer
Recommendation: enforce first, size second, with a dated re-measurement.
Why raising the limit is wrong as step one: the table is a symptom. Raising 4M to 8M moves the ceiling without removing the traffic, and the same 900k flows will simply consume the new headroom; CPU remains the real bottleneck because every session costs a lookup and an aging cycle, so users still see slowness. It also converts a visible alarm into a silent one - the next incident arrives with no early warning.
Controls to apply:
- Put the lab VLAN in its own zone or policy with an explicit deny to untrusted destinations, and cap it with a bandwidth/session limit so a lab mistake cannot exhaust a shared resource.
- Isolate and clean host 10.10.5.88; a host emitting 445 SYNs across two /16s is compromised or misconfigured, and blocking it is a security action, not a performance tweak.
- Rate-limit or block 445 outbound at the perimeter, which the policy set should already do.
- Verify NAT pool sizing and add addresses if port allocation is genuinely short; enable port-range allocation where supported to extend each address.
- Tune TCP aging only after confirming the 443 flows are idle rather than active - shortening aging on real long-lived application sessions causes user-visible drops. Use a dedicated long-connection rule for the applications that need it instead of a global timeout change.
Then re-measure the 09:00 peak for a week. If the normalised baseline still exceeds about 70% of the table or CPU stays above 70% at expected load, take that data to the capacity discussion - it is a defensible business case, unlike the current numbers.