High CPU after a temporary kernel debug session
An R81.20 open-server gateway has 8 CoreXL workers. After an engineer spent two hours troubleshooting a packet drop, users report slow throughput and one worker shows 99% CPU in top -H. The other workers are at 20-35%. cpview history shows the spike began during the troubleshooting window. The engineer used fw ctl debug -m fw + drop and captured output, but cannot remember whether debug was stopped. /var/log/messages is growing quickly with FW-1 debug messages. There is no disk pressure and no policy install in progress. fw ctl multik stat shows connections distributed normally across workers. A tcpdump sample shows no abnormal traffic mix. The gateway has 16 GB RAM and 4 GB free. The customer asks whether a reboot is required.
1. Which commands confirm whether kernel debug is still active, and how would you interpret them?
Read the worked answer
Start with fw ctl debug -info. It reports the active debug buffer and the modules and flags currently enabled, so you can see whether fw drop debug is still set. fw ctl debug -buf prints the buffer contents, which may show the old capture still being written. Check cpview CPU history and top -H to correlate the 99% worker with the debug window. Also inspect /var/log/messages for FW-1 debug lines. If fw ctl debug -info shows a non-empty mask for fw or drop, the debug is still active. The single hot worker is plausible if that worker is handling the traffic that matches the debug condition; fw ctl multik stat showing normal distribution does not rule out debug, because the debug cost is per matching packet.
2. What is the safe way to stop the debug and verify recovery, and why is a reboot not the first step?
Read the worked answer
Stop debug with fw ctl debug 0; this clears the debug flags rather than rebooting. Re-run fw ctl debug -info to confirm the flags are cleared, then monitor top -H and cpview for several minutes. If CPU returns to normal, the cause is confirmed. A reboot is not the first step because it is disruptive, may hide the evidence, and does not fix the process issue. If CPU remains high after debug is cleared, look for other causes: a traffic surge, a policy install, or a core dump. Capture cpinfo -z after recovery if you need support, but do not run it while the disk is under pressure.
3. What process change prevents this in a future case?
Read the worked answer
Adopt a debug checklist: record the exact fw ctl debug command, the start time, the expected duration, and the stop command; use fw ctl zdebug for short-lived captures where appropriate; and make fw ctl debug 0 plus fw ctl debug -info the final step before closing the ticket. Add a peer review or change record for kernel debug on production. Monitoring should alert on sustained per-worker CPU and on debug buffer growth. The goal is to make debug a bounded, reversible action rather than an open-ended session.