Two Interconnects, One Path: Asymmetric BGP and a Jumbo-Frame Backup

Start with gcloud compute routers get-status <router> --region=<region> on every router involved. It shows BGP session state, the MED received on each session, and the routes selected as best, which turns speculation into evidence.

Why this pattern appears:

• Attachment regions. A Cloud Router only accepts VLAN attachments in its own region. An Amsterdam connection and a Frankfurt connection therefore normally terminate on two different Cloud Routers in two different regions. With global dynamic routing, routes learned in one region are usable from another, but a resource prefers the route learned in its own region, so the attachment co-located with europe-west4 wins regardless of MED.
• Outbound (GCP to on-prem). Cloud Router's own advertised MED defaults to 0, so neither session offers the CE routers a preference signal; the CE routers break the tie themselves (often lowest router ID or first session established), which is why on-premises return traffic also piles onto one link.
• Inbound (on-prem to GCP). Cloud Router prefers the shorter AS path first and uses the received MED only to break ties among equal-length paths. If both sessions carry the same AS path and no MED difference, the choice is effectively arbitrary.

To make it deterministic:

1. On the on-prem CE routers, set local preference or MED so that each prefix has an intentional primary and backup path, and use AS-path prepending on the session you want to demote; prepending is visible across vendor platforms and survives Cloud Router's selection logic.
2. On the GCP side, set a higher MED on the session you want to demote where the CE advertises MED, and verify with get-status that the intended attachment is selected for each prefix.
3. Only then claim active/active. If you want both links used simultaneously, the MED values must be equal and the on-prem side must hash across them; otherwise document active/standby and size each link for full load.
4. Confirm symmetry per prefix rather than in aggregate. Asymmetric BGP is how you get one-way drops that look like a firewall problem.

The prefix is learnable only through the Frankfurt BGP session, so everything depends on that session advertising it and on the VPC making it available outside Frankfurt's own region.

Verification sequence:

1. On the Frankfurt Cloud Router, run gcloud compute routers get-status and confirm 10.22.0.0/16 appears as a learned route with the expected next hop and MED. If it is absent, the problem is on the CE: an outbound route filter, a missing network statement, or the prefix not being redistributed into BGP.
2. Confirm the VPC's dynamic routing mode is global. In regional mode, a route learned in Frankfurt is not usable by subnets in other regions, and europe-west4 traffic would simply have no path.
3. Check for a competing static route. A summary route such as 10.16.0.0/12 pointing at an NVA, or a leftover VPN route to the old datacentre, can match 10.22.0.0/16 with a lower priority number and black-hole the traffic even though BGP learned the prefix.
4. Check policy: an ingress firewall rule on the destination side, or a hierarchical deny, can drop the traffic after routing succeeds. Test with a Connectivity Test from a europe-west4 VM to a host in 10.22.0.0/16 before blaming BGP further.

Implementation risks to manage:

• The Amsterdam CE does not know 10.22.0.0/16. If MED or prepending steers europe-west4 egress over Amsterdam, packets for that prefix die on the Amsterdam path. Either advertise the prefix from both CEs, or confirm via get-status and traceroute that 10.22.0.0/16 is longest-prefix matched to the Frankfurt attachment while other traffic uses Amsterdam.
• Apply the change in a maintenance window with the backup path pre-verified, and keep the previous route-policy configuration recorded so you can roll back within minutes.
• After the change, retest return traffic: the on-premises side must have a route back to the europe-west4 subnet over the same or a valid path.

You get a PMTUD black hole: small requests work, 6 MB transfers hang.

MTU budgets on this design:

• Dedicated Interconnect supports 1500 or jumbo 8896, set on the VLAN attachment and on the VPC network.
• HA VPN tops out at 1460 bytes because of IPsec overhead.

On failover the path MTU drops from 8896 to 1460. A sending host discovers this only through ICMP "fragmentation needed" messages. Those ICMP messages are routinely filtered by on-premises firewalls and security appliances, and Google Cloud does not clamp MSS for you. The result is that the handshake and small responses succeed while the first full-size data segment is silently dropped and the connection stalls - which looks like an application bug under load and is often misdiagnosed as a database problem.

Prevention, in order of preference:

1. Use the lowest common denominator. Set the VPC network MTU to 1460 or 1500 so the cloud side never emits a packet the VPN cannot carry. Instances pick up the VPC MTU via DHCP; the change is not retroactive on running instances without a refresh, so schedule it. Keep the interconnect attachment and the VPC MTU consistent, and keep GKE cluster MTU consistent across node pools.
2. Clamp MSS on the CE routers. ip tcp adjust-mss 1360 (or the vendor equivalent) on the VPN-facing interface removes the dependency on PMTUD for TCP.
3. Permit ICMP type 3 code 4 end to end so PMTUD works for anything MSS clamping cannot cover, such as large UDP flows.

Validate before the failover test, not during it: from a VM, ping -M do -s 1432 toward on-premises to prove a 1460-byte path works, and repeat with a size just under the jumbo MTU on the Interconnect. Then run the failover in the monthly window with the MTU change already in place. Document that jumbo frames are an Interconnect-only optimisation, not a property you can keep "end to end" once an internet backup is in the failover path.