Warehouse Chassis Cluster Fails to Fail Over on an Upstream ISP Black Hole
Two SRX345s form a chassis cluster at a distribution warehouse. reth0.0 (untrust) and reth1.0 (trust) are in redundancy group 1, and node 0 is primary. The ISP handoff is a fiber-to-copper media converter plugged into ge-0/0/0 on node 0; ge-0/0/0 is a member link of reth0. At 02:10 the ISP upstream router stops forwarding traffic for 40 minutes, but the media converter copper port stays up and the SRX link LED remains lit. The cluster does not fail over. Users lose internet access until the ISP restores service. The control link and fabric link are healthy, and show chassis cluster status shows node 0 primary for RG1 the whole time. Redundancy group 1 monitors ge-0/0/0 and ge-0/0/1 with weight 255. The design uses no dynamic routing to the ISP; a static default route points at 203.0.113.1. The business wants sub-30-second failover to the secondary node ISP link, which is on a different provider.
1. Why did redundancy group 1 stay primary, and which show commands would prove that interface monitoring never fired?
Read the worked answer
Interface monitoring watches the link state of the member interface, not end-to-end reachability. The media converter kept the copper port up, so ge-0/0/0 never reported down; weight 255 therefore never decremented RG1. The static default route also remained installed because the next hop was still reachable at layer 2. Use show chassis cluster status redundancy-group 1 to confirm node 0 is primary with no failover count, show chassis cluster interfaces to see the monitored interface state, and show interfaces ge-0/0/0 terse to confirm link up. A packet capture or show route 0.0.0.0/0 on node 0 would show traffic still being sent to a dead ISP. The key judgement: a local link-up condition is not proof that the ISP path works.
2. What would you change to get sub-30-second failover, and what trade-off does that introduce?
Read the worked answer
Add IP monitoring to RG1, for example a probe to the ISP next-hop or a stable public target reachable through that ISP, with a weight of 255 and timers short enough to detect failure in under 30 seconds. On SRX chassis cluster this is configured under chassis cluster redundancy-group 1 ip-monitoring; it uses RPM-style probes. If the probe fails, RG1 fails over to node 1, whose ISP link is separate. The trade-off is false positives: a target outage or an ICMP-filtered path can trigger failover even when the ISP is fine. Use two targets, a target that is not ICMP-rate-limited, and hysteresis. Also confirm the secondary node ISP link is actually usable and that the static route exists on both nodes; otherwise failover just moves the outage.
3. What must you verify about NAT and sessions before promising that failover is transparent to users?
Read the worked answer
Session sync is only half the story. If source NAT uses the reth interface address, node 0 translates to ISP-A public address and node 1 will translate to ISP-B address. Existing sessions that survive sync point at the old public address, so return traffic from the internet may not match; TCP sessions can stall even though the cluster shows them as synchronized. show security flow session and show security nat source summary will show the translation. For transparent failover you need provider-independent addressing or a design that accepts session reset; otherwise tell the business that new sessions recover but existing TCP sessions may need to reconnect. Also check that the security policy, zones, and route are identical on both nodes.