Nightly Replication, Voice Gaps, and a Flapping HSRP Pair

They are the same underlying condition - congestion on a shared 1 Gbps trunk - but three different mechanisms, and that distinction drives the fix.

1. Voice gaps (3–8 s). OSPF's default dead timer is 40 s, so a reroute cannot produce a 3–8 s gap. HSRP failover is sub-second. What produces a multi-second audio gap is buffer occupancy: a 400 Mbps elephant flow plus voice in a single FIFO queue creates serialization delay and tail drop on the voice frames. 4% output drops on a 1 Gbps trunk is roughly 40 Mbps of discarded traffic - far more than voice consumes, which is exactly why voice suffers even though it is a tiny fraction of the load. This is a QoS problem, not a routing problem.

2. HSRP role changes. The hellos traverse the congested trunk. When they are dropped, each switch declares itself active; when congestion lulls and hellos resume, preempt on both sides forces the configured winner back, and the cycle repeats 6–10 times an hour. The tell is that the changes cluster inside the backup window and stop afterwards.

3. OSPF dead-timer expiry. Same mechanism, and the control-plane policer dropping in the OSPF class confirms that hellos are being discarded - possibly by the policer rather than the queue. Check show policy-map control-plane and the interface output queue drops together.

Evidence to pull before changing anything: per-queue drop counters, show policy-map interface, HSRP state history with timestamps, and show ip ospf neighbor to confirm whether adjacency resets precede or follow the HSRP transitions.

Order by what removes the clinical/business risk fastest, not by what is most elegant.

1. Shape the replication at the source. A policer/shaping policy at 250–300 Mbps with a defined burst stops the elephant flow from filling the trunk. This is a one-line change on one system and removes the root cause tonight. Policing alone is insufficient - a policer drops, a shaper queues and smooths; shape at the sender so the drop happens where the application can back off.

2. Fix the queueing. LLQ for voice (10% of 1 Gbps, i.e. 100 Mbps, is generous) and a guaranteed class for control-plane and routing protocols, with replication in a bandwidth-limited class using the remaining capacity. This buys protection when the shaper is misconfigured or a new backup job appears.

3. Stabilise the first hop. standby 10 preempt delay minimum 300 on the primary and remove preempt from the standby. Both-sides-preempt is the amplifier that turns hello loss into repeated outages. Better still, add object tracking so a degraded uplink yields a deliberate, damped failover rather than a flapping one.

4. Speed up and harden routing convergence. BFD at 300 ms for OSPF so a genuine failure is caught in under a second instead of 40 s, and confirm the control-plane policer has headroom for BFD plus OSPF.

5. Remove the shared fate. A second trunk, or a routed L3 path between the MDFs, so HSRP hellos and OSPF no longer depend on the same congested link as the backup job.

Step 1 buys the most for the least risk. Steps 2–4 prevent recurrence. Step 5 is the design fix that should go into the next change window.

The failure mode is a blackhole with no interface event.

ISP handoffs are frequently L2 or Ethernet-presented: the local interface stays up even when the provider's path is dead. A floating static is withdrawn only when the interface or next hop goes down. With a silent upstream failure, the router keeps advertising and using the dead default, AD 250 never matters, and traffic is blackholed. Users see a total outage while the router reports healthy interfaces - the worst possible combination for troubleshooting.

Closing it:

• Track the default with IP SLA: probe a stable destination beyond the ISP, and bind the static to an object with track. On probe failure the static is withdrawn and the other default is used. Choose the probe target carefully - a public DNS resolver or the ISP's next-hop-plus-one, and budget the detection time (probe interval x threshold) into the failover expectation.
• Set AD so the paths are deterministic: primary default lower AD, backup higher. Equal AD on both creates ECMP or asymmetric routing to the internet, which breaks anything stateful.
• Track the same object into HSRP so the LAN first hop follows the same decision the routers make; otherwise the gateway points at a router that has already given up on its uplink.
• If they intend to stay dual-homed, move to eBGP with provider-assigned addresses and prefix filtering - it detects path loss without a synthetic probe and scales to a third provider.

Judgement call: IP SLA tracking is the right answer for one or two providers and a small team; BGP is the right answer if the internet path is business-critical or a third circuit is likely. Document the detection time either way - a silent blackhole that takes 30 s to detect is still an outage.