Policy installation fails on one cluster member after /var/log fills
An R81.20 ClusterXL HA pair (gw1 active, gw2 standby) is managed by mgmt-a. A routine policy install to the cluster finishes successfully on gw1 but fails on gw2 with 'Installation failed. Reason: Failed to connect to remote peer'. SmartConsole shows the cluster as OK overall but gw2 SIC status as 'Unknown'. On gw2 console, fw stat returns 'Can't connect to cpd'; cpwd_admin list shows cpd not running and the watchdog restarting it every few minutes. df -h shows /var/log at 100% (6.9G of 7.0G). /var/log/cpwd.elg ends with 'No space left on device'. cpinfo -z cannot complete because it tries to write a temporary file. The last known good policy on gw2 is three weeks old; gw1 is carrying all traffic. No disk changes were made in the last 24 hours, but debug files and old core dumps are present under /var/log.
1. What is the most likely root cause, and which evidence proves it rather than a SIC/certificate problem?
Read the worked answer
The strongest hypothesis is that /var/log is full, which prevented cpd from writing and caused it to crash; the SIC 'Unknown' is a symptom, not the root cause. The evidence chain is: df -h shows /var/log at 100%; cpwd_admin list shows cpd not running despite watchdog attempts; /var/log/cpwd.elg records 'No space left on device'; and fw stat cannot connect to cpd. A SIC or certificate problem would normally leave cpd running and produce SIC messages in cpwd.elg or sic.elg, not disk errors. Because gw1 installs and gw2 does not, and gw2 cannot talk to cpd, the management server cannot complete the install. cpinfo -z failing due to a temporary-file write error is consistent with the same full filesystem. The stale policy on gw2 is a risk if gw1 fails over.
2. How would you restore cpd safely while preserving evidence and avoiding cluster failover risk?
Read the worked answer
Do not reboot first and do not delete active logs blindly. Capture df -h, df -i, du -x /var/log | sort -h, and a file listing for evidence. Identify old rotated logs, debug captures, and core dumps that can be moved to external storage or compressed; preserve current logs and the cpinfo output. Follow the customer's Check Point log-retention procedure rather than truncating live files. After freeing enough space, restart cpd with cpwd_admin start -name CPD -command cpd if the watchdog has not already succeeded, then verify with cpwd_admin list and fw stat. Check df -h again to ensure headroom. If the cluster is at risk because gw2 is stale, consider a controlled failover only after cpd and SIC are healthy, not as the first fix.
3. What must be verified before the next policy install, and what prevents a repeat?
Read the worked answer
Before the next install, verify that cpd is running; fw stat shows the expected policy name; SIC is trusted from SmartConsole; /var/log has at least 20% free and inode headroom; log rotation and core-dump retention are configured; and monitoring alerts on filesystem thresholds. Install to gw2 first, or to the cluster while watching both members, and confirm the policy name and install log on each. The prevention is capacity and retention management plus an alert before 90% full, because a full /var/log will reproduce the same failure. If core dumps are the large files, collect the required support evidence before removing them, and fix the underlying process crash so the dumps stop recurring.