OCI Multicloud Architect Associate study materials
OCI Multicloud Architect Associate study resources

Overlapping CIDRs, Two Hubs, and a Redundant Circuit That Made Things Worse

OCI Multicloud Architect Associate · Put your knowledge to work in context.

All case studies

Overlapping CIDRs, Two Hubs, and a Redundant Circuit That Made Things Worse

Contoso Financial has an OCI VCN of 10.20.0.0/16 in US East (Ashburn) hosting an Autonomous Database with a private endpoint and an OCI GoldenGate deployment. Its Azure estate in East US was built from the same IP plan and uses 10.20.0.0/16 for the hub VNet, with 10.20.16.0/20 assigned to AKS node pools and 10.20.32.0/20 to internal load balancers. Oracle Interconnect for Azure already links the two clouds in these regions. On-premises (10.0.0.0/8) reaches OCI over a 10 Gbps FastConnect and Azure over ExpressRoute, terminating on separate customer routers that share a single BGP ASN. Requirements: AKS pods must reach the Autonomous Database private endpoint without touching the internet; on-premises must reach both clouds; the acquired business unit on 172.16.0.0/12 must reach only its own Azure subscription. After a second FastConnect circuit was added last month for redundancy, about 5% of connections from Azure to the OCI database time out. Failures cluster in the same 15-minute window each hour, and large result sets fail far more often than small queries.

1. A junior engineer proposes VCN peering between the OCI VCN and the Azure hub VNet. Explain why that cannot work and describe the correct connectivity model.

Read the worked answer

VCN peering is an OCI-to-OCI construct: it links two VCNs whose traffic is carried inside OCI's network, and remote peering does the same across OCI regions through DRG. Neither can terminate on an Azure VNet, because Azure VNet peering is a separate product with separate route semantics. There is no cross-cloud peering primitive.

The correct model has three layers:

  • Transport: Oracle Interconnect for Azure pairs a FastConnect virtual circuit on the OCI side with an ExpressRoute circuit on the Azure side, in regions that have a published interconnect pairing. Order both halves; they are one logical link but two provider tickets.
  • OCI aggregation: the FastConnect virtual circuit attaches to the DRG. The DRG owns the route tables that decide which attachment a prefix is learned from and advertised to. VCNs attach to the DRG as separate attachments.
  • Azure aggregation: the ExpressRoute circuit connects to an ExpressRoute gateway in the hub VNet; spokes reach the hub through VNet peering, and the gateway transit setting lets spokes use it.

Traffic then flows VCN subnet -> DRG -> FastConnect/ExpressRoute -> Azure hub gateway -> peered spoke. On-premises arrives on its own FastConnect to the same DRG, so the DRG is also where you control whether Azure and on-premises can talk to each other - by default you should not advertise Azure prefixes to on-premises unless required, which is a governance decision as much as a routing one.

2. Both sides use 10.20.0.0/16. What are the realistic remediation options, and which do you recommend?

Read the worked answer

Overlapping prefixes cannot be routed across the interconnect at all: the DRG and the ExpressRoute gateway will each see a prefix they believe is local, and packets will be dropped or black-holed rather than translated. OCI NAT Gateway is outbound-only and provides no inbound translation for transit traffic, so 'just NAT it' is not available as a platform feature.

Options, in order of long-term quality:

  1. Re-IP the Azure hub VNet. Because AKS node pools are the main consumer, this can be done by building a new node pool on a non-overlapping range (for example 10.60.0.0/20), cordoning and draining the old pool, and moving internal load balancers to new frontend IPs. Pods on an overlay network are unaffected. Cost: a maintenance window and DNS changes.
  2. Front the database with a stable, non-overlapping access point - a private load balancer or a small NVA performing DNAT in Azure - so consumers address a VIP rather than the overlapping range. This works, but it hides the problem and adds a stateful hop you must scale and patch.
  3. Re-IP the OCI VCN instead. Usually worse, because the database private endpoint, GoldenGate, and any downstream consumers all move.

Recommend option 1 with option 2 only as a temporary bridge for the handful of consumers that cannot move quickly. Whichever you choose, add an IP address management control: an allocation register owned by the network team, with a mandatory check in the landing-zone request process, so the next acquisition does not repeat this.

3. Diagnose the intermittent timeouts. What is the most likely mechanism, and how would you prove it rather than guess?

Read the worked answer

Two signals point to different mechanisms, and the answer should separate them.

Signal 1 - large result sets fail more often. That is the signature of an MTU / path-MTU discovery black hole. Azure VNet interfaces are 1,500 bytes but the ExpressRoute path may not carry full-size frames end to end; if ICMP fragmentation-needed messages are filtered anywhere, small queries succeed and large responses hang. Prove it with ping with the do-not-fragment bit set at 1,472 / 1,400 / 1,380 bytes from a pod to the database endpoint, and by capturing on both ends to see retransmissions with no ICMP response. Mitigation: clamp MSS on the Azure side or set the tunnel/interface MTU, and stop blocking ICMP type 3 code 4.

Signal 2 - a repeating 15-minute window each hour. That is almost certainly a scheduled workload or a control-plane event, not random loss. Correlate the window against OCI and Azure backup jobs, GoldenGate extracts, and monitoring scans; then check interface counters on both FastConnect circuits for errors, discards, and throughput saturation in that window. Also check BGP: with two circuits and a shared ASN, verify that each circuit has a stable session and that the DRG route table and Azure route tables agree on the preferred path. Asymmetric routing - outbound via circuit A, return via circuit B - produces exactly this kind of partial failure.

Prove it with data: per-circuit byte and error counters, BGP uptime and flap history, DRG route table dumps from both sides, and a packet capture at the database endpoint. Change nothing until the capture shows which of the two mechanisms is real.