Circuit B Was Not Idle by Design: FastConnect/ExpressRoute Failover on Oracle Interconnect for Azure
Solstice Retail runs its order platform in OCI eu-amsterdam-1 (VCN 10.20.0.0/16) and its storefront APIs and analytics in Azure West Europe (VNet 10.40.0.0/16). The clouds are joined by Oracle Interconnect for Microsoft Azure using two 10 Gbps FastConnect virtual circuits, A and B, each mapped to its own ExpressRoute circuit, both recorded as terminating on the same DRG. An internet IPsec VPN is the documented backup path. Peak interconnect traffic is 6.8 Gbps. During a carrier maintenance window circuit A was down for 40 minutes. Azure-to-OCI traffic continued at full rate, but OCI-to-Azure throughput collapsed to roughly 300 Mbps and 12% of order-submission calls timed out. The VPN carried nothing. Post-incident review shows circuit B had moved almost no bytes in the previous 90 days, the DRG route table listed a single next hop for 10.40.0.0/16, and the Azure VNet gateway SKU is ErGw2Az.
1. Explain why circuit B carried no traffic for 90 days and give an ordered troubleshooting sequence to confirm the cause.
Read the worked answer
Two things must be separated: a healthy design where traffic is legitimately spread per flow, and a broken design where one circuit is simply not in the routing table.
Zero bytes over 90 days is anomalous. ExpressRoute and the DRG hash flows across equal-cost paths, so even an idle-looking circuit should show traffic from thousands of independent order flows. The DRG route table with a single next hop for 10.40.0.0/16 is the smoking gun: OCI was only aware of one path, so failover had nothing to fail over to.
Ordered checks:
- DRG route table for 10.40.0.0/16 - how many next hops, which attachments. One hop means a missing or rejected advertisement.
- BGP state per virtual circuit: is the session Established, are prefixes received and advertised on both, is there a prefix limit or a route-map/route-policy dropping the Azure prefix on B?
- Attachment topology: confirm both virtual circuits attach to the same DRG and are imported into the same DRG route table. A second DRG without peering produces exactly this symptom.
- Azure side: both ExpressRoute circuits connected to the same ExpressRoute gateway, equal connection weight, and effective routes on a NIC showing both circuits.
- Compare AS path length and MED - prepending on B during an old test that was never reverted is a common leftover.
- Only after routing is symmetric should you discuss hashing or per-flow behaviour.
2. The CTO wants maintenance on either circuit to be invisible. Describe the routing and monitoring design, and state why the VPN backup did not engage.
Read the worked answer
Design for two independent goals: capacity survival and deterministic path selection.
Capacity first. A single circuit must carry the full 6.8 Gbps peak, plus burst headroom. Both circuits are 10 Gbps, so the FastConnect side is adequate, but the Azure ExpressRoute gateway SKU is not: ErGw2Az tops out around 2 Gbps aggregate, so even with both circuits healthy Azure could never accept the design throughput. The gateway must move to ErGw3Az (10 Gbps class), or traffic must be split across VNet gateways. This also explains part of the collapse to 300 Mbps - after A failed, Azure accepted only what its gateway could absorb.
Path selection. Either run active/active with equal AS path and equal Azure connection weight so per-flow hashing spreads load, or run an explicit primary/secondary using Azure connection weight plus AS path prepending on circuit B toward Azure and a higher local preference on A toward OCI. Never leave it implicit: an implicit design is what produced a 90-day idle circuit.
Monitoring. Alarm on BGP session state per circuit, per-circuit byte counters with a floor alarm (not just a ceiling), synthetic probes sourced from each cloud that measure loss and RTT, and a route-count check that fires when the DRG sees fewer than two next hops for any interop prefix.
The VPN stayed silent because a tunnel being up is not a route. The DRG route table had one next hop, so the VPN attachment was never a candidate; the backup path needed its own DRG route table entry, dynamic routing over the tunnel with a longer AS path, and a periodic failover test that proves bytes actually traverse it.
3. Six months later security asks to insert a stateful firewall between the clouds and to keep both circuits active. What breaks, and how do you reconcile the two requirements?
Read the worked answer
A stateful firewall expects to see both directions of a flow. With two equal-cost circuits and per-flow hashing done independently in each direction, the return packet can land on the other circuit and arrive at the firewall with no session entry - the connection is dropped and, worse, the failure is intermittent and load-dependent, so it looks like an application problem.
Reconciliation options, in order of preference:
- Keep stateful inspection, make the path symmetric. Pin interop traffic to one circuit per direction using routing policy (prepend on B toward Azure, preference on A toward OCI), and keep the second circuit as a standby that is proven by scheduled tests. You lose active/active bandwidth but keep firewall semantics and deterministic behaviour.
- Use a firewall cluster in a transit VCN where both circuits terminate on the same inspection nodes, with session synchronisation between cluster members. Asymmetric returns are then handled inside the cluster rather than by the routing fabric, because both paths land on the same device pair.
- Replace stateful inspection on the interop path with stateless controls (security lists, NSGs on both sides, flow logs, TLS with mutual authentication) and keep stateful inspection only for internet ingress.
Whichever you choose, verify the firewall's throughput against the 6.8 Gbps peak - a pair of 2 Gbps appliances becomes the new bottleneck and the new single point of failure. Also re-test the ErGw3Az gateway throughput with inspection inline, since inspection adds per-flow latency that changes TCP behaviour at peak load.