A Backup Default Route That Never Activated

Preference only arbitrates between routes that are actually present in the routing table - it is not a health check.

Three things had to be true for failover, and none were:

1. The primary route had to be removed from the table. It was not. From r1's perspective nothing failed: ge-0/0/0 stayed up (the carrier's PE kept the link alive), so the next hop 203.0.113.1 remained directly reachable via the connected subnet 203.0.113.0/30.
2. A static route has no liveness detection by default. Junos never asks whether the next hop can actually forward traffic - only whether it is resolvable. It was, so the route stayed installed and active.
3. The backup could only become eligible once the preference-5 route left the table. Since that never happened, the preference-20 route stayed inactive for the whole 50 minutes.

ISP-A's router was alive and still answering on the link; it had simply lost its upstream. That failure is completely invisible from r1 at Layer 2 and Layer 3 next-hop resolution. r1 kept forwarding faithfully into a black hole.

Confirming the state:

• show route 0.0.0.0/0 detail - shows the active next hop, the preference, and the second path marked Inactive reason: Route Preference. Seeing both paths present proves the backup was eligible but out-ranked, not missing.
• show route forwarding-table destination 0.0.0.0/0 - proves which next hop the PFE actually has programmed, ruling out a control-plane/forwarding-plane mismatch.
• show route 203.0.113.0/30 - confirms the next hop is still directly reachable, i.e. the route was never withdrawn.
• show interfaces ge-0/0/0.0 extensive - confirms link state up, no interface drops, matching the "link stayed up" observation.

Why pinging 203.0.113.1 is misleading: that address belongs to the ISP's PE router, which was healthy the whole time. ping 203.0.113.1 would have succeeded throughout the outage and produced a false green. The only meaningful reachability test is through the ISP to a destination beyond it, sourced from r1 or from a downstream host - which is exactly what an RPM probe or an external monitoring agent does and what a next-hop ping does not.

Option 1 (preferred): BFD on the static next hop.

set routing-options static route 0.0.0.0/0 next-hop 203.0.113.1 \
    bfd-liveness-detection minimum-interval 300 multiplier 3

Junos withdraws the static route when the BFD session to 203.0.113.1 fails, which removes the preference-5 route from the table and lets the preference-20 backup become active within roughly a second. Dependency: ISP-A must run or at least answer BFD on that link - this is a contractual/ticketing item with the carrier, not something you can turn on unilaterally and assume.

Option 2: run BGP with each ISP and use BFD. The default prefix disappears when the session drops, and BFD makes session-down detection sub-second rather than timer-bound. This is the most robust option because it tracks end-to-end protocol state, not just link state. Dependency: ISP cooperation on ASN/session configuration.

Option 3: probe-based route withdrawal. Use RPM probes to a destination beyond each ISP and, on current Junos, services ip-monitoring to withdraw the primary route when the probe fails. Handles the "ISP alive but not forwarding" case without carrier BFD support. Dependencies: choosing a probe destination that is not itself ICMP-filtered (a blocked probe produces a false failover - a self-inflicted outage), and tuning thresholds so you do not flap on transient loss.

Independent hardening: keep the backup preference well separated from the primary so there is no ambiguity, and add external (off-net) monitoring of an end-to-end path - an internal-only monitor cannot see this failure mode at all.