Intermittent Outages After Adding a Second Hybrid Circuit
After adding a second 10 Gbps Direct Connect circuit between an on-premises data center and AWS, users report 3-5 second outages every 20-30 minutes. The on-premises network has two routers running eBGP with AWS virtual private gateways over the two circuits. Each router advertises 120 prefixes, including a summary 10.0.0.0/8, and receives 80 AWS prefixes. The outages began after the second circuit was cut over. Monitoring shows BGP session resets on one circuit, and during the outage, traffic shifts to the other circuit but then returns, causing asymmetric routing. The AWS side uses default BGP timers (hold time 90 seconds), and the on-premises routers use hold time 30 seconds. The on-premises team recently enabled BFD on one router only. There is no route map filtering the 10.0.0.0/8 summary, and the AWS VPC route tables have static routes pointing to the virtual private gateway. The business impact is high because the outage affects order processing. The team needs a stable design and a troubleshooting sequence.
1. What is the most likely root cause of the BGP resets, and what immediate mitigation would you apply?
Read the worked answer
The combination of BFD enabled on only one router, mismatched BGP hold timers (30 seconds on-premises versus 90 seconds on AWS), and a broad 10.0.0.0/8 summary is the likely cause. BFD on one side can cause the session to be torn down when BFD packets are not symmetrically processed. The summary route can blackhole or shift traffic when one circuit resets, and the 30-second hold time makes the on-premises router declare the session down faster than AWS. Immediate mitigation: disable BFD on the single router or enable it consistently on both routers and both circuits; align BGP timers to the same values on both sides, preferably 30/90 or 10/30 with BFD; and remove or filter the 10.0.0.0/8 summary so more-specific prefixes are advertised. Then clear the BGP session during a change window and monitor for 24 hours.
2. Propose a stable BGP and BFD configuration for both circuits, including route policy.
Read the worked answer
Use the same BGP timers on both on-premises routers and both AWS virtual private gateways. Enable BFD on both routers and both circuits, or disable it everywhere; do not run asymmetric BFD. Advertise only the specific prefixes that AWS needs, not 10.0.0.0/8. If a summary is required, advertise it with a route map that sets a lower local preference or longer AS path than the more-specific routes, and ensure AWS has a valid more-specific path. On-premises, use route maps to set local preference so the primary circuit is preferred and the secondary is backup, but allow both to carry traffic if equal cost is desired. Use maximum-prefix limits and graceful restart. On AWS, use the virtual private gateway or transit gateway with equal-cost multipath if both circuits should be active. Document the AS numbers and prefix filters.
3. How would you eliminate asymmetric routing and verify the fix?
Read the worked answer
First, make both on-premises routers advertise the same prefixes with consistent BGP attributes. If both circuits are active, use equal-cost multipath and ensure the return path from AWS uses the same circuit by setting local preference or AS path prepending consistently. Remove static routes in VPC route tables that point to a single virtual private gateway if they override BGP-learned routes; use route propagation instead. For active/backup, set a higher local preference on the primary on both sides. Verify with show ip bgp, show ip route, and traceroute in both directions. Compare the forward and return paths for a sample of prefixes. Check AWS Direct Connect virtual interface BGP status and VPC flow logs for asymmetric flows. Run a 24-hour soak test with periodic circuit failures and confirm no route flapping and no 3-5 second outages.