JNCIA-Junos: Eight Gigabits Forwarded, Pings to the Router Dropped

Eight Gigabits Forwarded, Pings to the Router Dropped

A Junos MX-series router forwards about 8 Gbps of transit traffic between two provider handoffs. Over the last 20 minutes, operators report that SSH to the router takes seconds per keystroke, and pings addressed to the router's own ge-0/0/0.0 address show 40% loss and 250 ms latency. Pings through the router, from a server behind it to a server beyond it, show 0% loss and normal round-trip time, and customer traffic is unaffected.

show chassis routing-engine reports 97% CPU. show interfaces extensive on both provider interfaces shows no input or output drops. The router runs eBGP to two ISPs with full tables and OSPF to the internal core. Two hours ago an engineer enabled traceoptions with flag all on BGP.

1. Explain how the router can forward 8 Gbps while failing to answer pings addressed to itself.

Read the worked answer

This is a control-plane problem, not a forwarding problem.

Junos separates the two planes:

Forwarding plane (PFE / Trio): transit packets are switched in hardware using the forwarding table the RE has programmed. They never touch the Routing Engine, so a saturated RE cannot slow them down.
Control plane (Routing Engine): rpd, the kernel, management processes, protocol state and all traffic addressed to the device itself.

Packets destined to ge-0/0/0.0 must be punted from the PFE over the host path, queued, and serviced by the RE kernel. When the RE is pinned at 97% by a control-plane process, that host-bound queue overflows and ICMP/SSH are dropped or delayed, while forwarded traffic stays clean.

Pings through the router look perfect because those packets are forwarded by the PFE and are never answered by the RE. The symptom pair (device-addressed traffic degraded, transit traffic clean, RE CPU pegged) is the classic signature of a control-plane overload.

2. Give the commands you would use to confirm the diagnosis and find the trigger, and say what each proves.

Read the worked answer

show chassis routing-engine - confirms RE CPU utilisation and which RE is hot.
show system processes extensive - a top-style view; identifies whether rpd, the kernel, or another daemon owns the CPU (traces are typically written via the kernel/rpd path).
show task and show task io - rpd thread-level view; shows whether rpd is spinning on protocol work or on logging.
show log messages | last 200 - look for BGP trace output and session churn; a flood of trace lines is itself the evidence.
show bgp summary and show ospf neighbor - establishes whether the churn is real (flapping sessions, constant SPF) or purely trace overhead.
show configuration | compare rollback 1 - correlates the start of the problem with the traceoptions change; the timing (two hours) is the strongest clue.
monitor traffic interface ge-0/0/0 no-resolve - shows what is being punted to the RE.

Fix first, tune second: deactivate protocols bgp traceoptions, commit confirmed 5, confirm, then re-check CPU. Tracing with flag all against a full table can generate enough log volume to saturate the RE even without any protocol instability. If churn is genuine, add control-plane protection (loopback filter with policers, and ddos-protection/screen on MX/SRX) before re-enabling any trace with a narrow flag and a file size limit.

3. A colleague suggests the RE CPU is a red herring and the drop must be in the PFE. How would you test that hypothesis, and what would you look for if the RE were idle while transit traffic dropped?

Read the worked answer

Test the hypothesis by checking whether forwarding itself is degraded, independently of the RE:

show interfaces extensive | match "drops|errors" on both provider links - distinguishes interface-level discards from host-path discards.
show interfaces queue ge-0/0/0 - CoS output drops indicate congestion/queueing rather than a lookup problem.
show pfe statistics traffic (MX) - look specifically at the host path counters; they will be incrementing here even though forwarding counters are clean.
show route forwarding-table destination <a customer prefix> - confirms the PFE has the expected next hop programmed (a missing/stale entry would cause PFE-side drops with an idle RE).
show firewall - confirms whether a filter or policer is counting discards.
show chassis alarms, show chassis fpc, show log messages - hardware/optic faults and FPC restarts.

If the RE really were idle and transit dropped, the suspect list shifts to: a stale or missing forwarding-table entry, a CoS policer, an MTU/fragmentation mismatch on one path, a firewall filter applied in the wrong direction, a hardware/optic fault, or an asymmetric path where return traffic is dropped elsewhere. The discriminator is simple: host-path counters climbing with clean interface counters points at the RE; interface or queue counters climbing with an idle RE points at the PFE or the path.