OSPF Flapping After a Core Upgrade

Replacing the LAGs is premature. The symptom pattern (regular 10-15 minute flapping, correlated with a software change on one side) points more strongly at control-plane or physical-layer causes than at LAG hardware.

Differential diagnosis:

• Software/version mismatch: the new core runs a newer release. Adjacency flapping after a one-sided upgrade is a classic signature. Check for known issues, changed default timers, or changed LAG/OSPF interaction in the release notes.
• Physical layer: a marginal optic or dirty connector can cause intermittent link loss on one member of a LAG, which can trigger reconvergence. Check interface error counters, CRC, and optic diagnostics on both sides.
• LAG negotiation: if LACP is in use, check that both members are bundled on both sides and that no member is suspended and re-added. A flapping member changes effective capacity and can disturb OSPF over the LAG.
• MTU mismatch: a mismatch can drop OSPF hellos or cause adjacency resets under certain conditions.
• Timers and load: if hellos are being dropped under voice load, check for control-plane policing or queue drops on the core.
• Duplex/speed negotiation: auto-negotiation mismatches on a replacement port.

First moves: capture interface counters and optic levels over a flap window, compare OSPF hello/dead timers on both sides, and check the release notes for the new core. Only replace hardware if the counters implicate a specific member.

Control-plane evidence:

• OSPF neighbor state transitions with timestamps, and the reason code if available (hello timeout, dead interval, interface down).
• Syslog and trap timeline from both core and distribution, aligned to a common clock (verify NTP sync first).
• Interface state changes (up/down, LACP member add/remove) on every LAG member during the window.
• CPU and control-plane policing counters on the core during the flap.

Data-plane evidence:

• Interface error counters (CRC, input errors, runts, giants) and optic receive power on each LAG member.
• Queue drops on the EF queue during the window, to see whether voice load is correlated with the flap.
• Flow or packet capture at the core showing whether voice packets are dropped only during reconvergence or continuously.

Distinguishing logic:

• If OSPF transitions align with LACP member state changes or interface errors, the cause is below OSPF (physical or LAG).
• If OSPF transitions occur with clean interfaces and no LACP events, look at control-plane policing, MTU, timers, or software behavior.
• If voice drops occur without OSPF transitions, the problem may be data-plane (queueing) rather than routing convergence.

Clock discipline: without synchronized timestamps, correlating events across devices is guesswork. Confirm NTP before drawing conclusions.

Safe mitigations (reversible, low blast radius):

• Pin OSPF to a single, verified-healthy LAG member or reduce the LAG to one member temporarily, if the flap is isolated to one member. This narrows the failure domain without a software change.
• Increase OSPF hello/dead timers moderately to ride out short interruptions, accepting slower detection. Document the trade-off.
• If a specific optic or port is implicated, replace that optic or move the link to a spare port.
• Apply a temporary route preference or static backup for the affected buildings so voice does not depend on the flapping adjacency.

Risky mitigations (avoid during business hours unless necessary):

• Downgrading or upgrading the core software mid-day. This is the likely root-cause fix but carries the largest blast radius and needs a window.
• Replacing LAG hardware without evidence, which may not fix the issue and introduces new variables.
• Broad timer changes across the whole campus, which can mask the problem and affect convergence everywhere.

Plan: apply the reversible mitigation, keep collecting evidence, and schedule the software alignment in a maintenance window with a rollback plan. Verify after each step that voice drops stop, so you know which change mattered.