CCTE R81.20 cheat sheet

Layered isolation order: confirm symptom scope (one client, one subnet, all users) → verify object/policy correctness → verify install status → verify Gateway daemons → verify packet path (inspection, NAT, routing) → verify logging/export.

cpstat reads live daemon and blade status on a Security Gateway: cpstat os, cpstat fw, cpstat -f all fw; use it before restarting anything.

cpinfo -o <file> bundles OS, product versions, policy install state, and logs; required baseline artifact for support escalation and version comparisons.

fw stat shows installed policy name and install time per target; a mismatch between SmartConsole install status and fw stat output indicates a partial or failed install.

cpwd_admin list shows process state (UP/DOWN/EXITED) and restart counters per daemon; repeated EXITED entries point to a crash loop, not a config error.

Log level escalation ladder: fw ctl debug / fw ctl zdebug for kernel-level, fw debug for user-space FW daemons, and per-blade debug flags in SmartConsole for inspection blades.

zdebug output is not persisted to disk; capture with fw ctl zdebug -m <module> + <topic> | tee file and always stop the debug (- instead of +) when finished.

Change control rule: capture cpinfo, current policy name, and fw stat output before any change so rollback targets are explicit.

Management daemons: cpm (main management process), cpd (daemon supervisor and API), fwm (legacy policy/object worker), CkpSSLSessionHandler, and postgres/solr for the data layer.

cpstart/cpstop restart all management processes; prefer targeted cpwd_admin stop/start for a single daemon to avoid interrupting connected GUI and API sessions.

api status reports the Management API server state; if the API is down, SmartConsole features that rely on it (some views, revisions) fail while legacy object operations may still work.

Revision control: every install creates a revision; show-revisions and revert-to-revision via API or the Install Policy window restore object/policy state, not Gateway state.

SmartConsole login failures split into: certificate/trust (SIC or ICA), user authentication (local vs LDAP/AD), and permission profile - check $FWDIR/log/fwm.elg and $CPDIR/log/cpd.elg.

Multi-Domain: MDS-level objects and per-domain objects are separate; a global object change requires install to the Domain Management Server, not the MDS.

Database consistency: cpdb -r style repair and migrate export/import are the supported paths; never hand-edit object tables in the underlying database.

$FWDIR/conf/objects_5_0.C is a legacy representation; use dbedit or the API for object changes so validation and revision tracking apply.

Install chain: policy compiled on Management (fwm) → transferred to Gateway → verified → installed by fw process; failure at any step leaves the previous policy active.

fw fetch/fw pull pulls policy manually; fw stat then confirms whether the new policy name is active before assuming the change took effect.

SIC failure symptoms: cpstart errors, cpd down, or install error "SIC status not initialized"; diagnose with cpconfig SIC reset or sic troubleshooting in $CPDIR/log/cpd.elg.

Policy install vs. blade activation: enabling a Software Blade requires a policy install to activate it on the Gateway; blade status shows in cpstat only after install.

Cluster members: policy installs to all members; a member in "Ready" but not "Active" state does not forward traffic - check cphaprob stat and cphaprob -a if.

cphaprob list enumerates cluster and Gateway critical processes; a failing critical process triggers failover in HA clusters.

Rule order matters for both accept and drop: an earlier drop/block rule shadows later accept rules; verify with the policy hit count and the log's rule number.

NAT and policy are separate install artifacts; a policy install pushes both, but manual fw operations may leave NAT out of sync.

fw monitor captures at four inspection points: i (inbound pre-inspection), I (inbound post-inbound-chain), o (outbound pre-outbound-chain), O (outbound post-inspection); the point where packets disappear localizes the failure.

fw monitor -e "accept host(10.0.0.1) and port(443);" filters at capture time; without a filter, high-traffic Gateways generate overwhelming output.

tcpdump sees the interface before inspection; fw monitor sees the kernel inspection path. A packet in tcpdump but not in fw monitor indicates a pre-inspection drop (routing, interface, or anti-spoofing).

Anti-spoofing drops appear as "Address spoofing" in logs; verify the topology object's interface leads and the routing table before changing the topology.

Logging: fw logswitch rotates logs; fw log -n tails recent log records; fwm logexport converts binary logs to text for offline analysis.

Log server unreachable → Gateway buffers logs locally; check $FWDIR/log/fwd.elg and the Log Server SIC/connectivity before assuming the event was not logged.

Inspection debugging: fw ctl debug -m fw + drop shows drop reasons in the kernel; fwaccel stats -s shows SecureXL accelerated vs. slow-path counts.

SecureXL: a packet handled in the fast path may not appear in fw monitor; use fwaccel off temporarily to force the slow path when debugging.