Replacing a Failed Cluster Node Without Forcing Failover
An NGFW cluster at a factory has two nodes in active/standby. Node 2's system disk fails. The spare has the same NGFW software version, but its NGFW ID is new. The admin creates a new one-time password for node 2 in the existing cluster object, runs initial configuration, and the node appears in the SMC as 'Inactive'. The cluster's heartbeat links are up and node 1 is active. The admin plans to run sg-toggle-active on node 2 to force it online, then install the policy. The factory's production line runs 24/7; a wrong failover could stop traffic. Backups of SMC configuration are 12 hours old.
1. Why is forcing node 2 active with `sg-toggle-active` the wrong next step?
Read the worked answer
An inactive node has not synchronized the cluster configuration, policy, or state, and may not have validated interfaces, routing, and licenses. Forcing it active can move production traffic to a node that is not ready to forward, causing an outage rather than restoring redundancy. The toggle command is intended for a controlled switchover between healthy cluster members, not for initializing a replacement node. The correct goal is to bring node 2 to a synchronized standby state while node 1 remains active. Only after synchronization and validation should a planned switchover be tested in a maintenance window.
2. Outline the correct sequence to return node 2 to a healthy standby state.
Read the worked answer
Confirm that the replacement node's identity and one-time password match the node object, and verify the software version and licenses. Complete initial configuration, then use the SMC's node recovery or synchronization process to pull the cluster configuration and policy from the active node. Check that the heartbeat links, CVI/NDI addressing, interfaces, and routing are correct, and that the node reports the expected policy version. Install or refresh policy as required, then monitor cluster status until node 2 is a synchronized standby. Keep node 1 active; test a controlled switchover later, outside production hours, and switch back if needed.
3. If node 2 remains inactive after recovery, what would you check before considering a different replacement or restore?
Read the worked answer
Check the heartbeat interfaces and cabling, CVI/NDI reachability, interface link state, and whether both nodes agree on cluster membership. Verify software and policy version compatibility, licenses, disk health, and system logs on both the node and the SMC. Confirm that the node object's NGFW ID and one-time password match the replacement, and that no stale node entry is conflicting. If synchronization still fails, do not force active; instead replace the node again or restore from a known-good SMC backup with vendor support. The priority is preserving the active node and production traffic, not shortening the maintenance window by skipping validation.