Unhealthy Backends, a Silent NVA Route, and a Subnet That Cannot Grow

The contrasting results are the clue: the bastion's curl proves the application path, not the prober path.

A bastion curl originates inside the VPC (10.0.0.x) and matches the blanket 10.0.0.0/8 rules. Load balancer health checks do not come from inside the VPC - they come from Google's prober ranges (35.191.0.0/16 and 130.108.0.0/16, plus 209.85.152.0/22 and 209.85.204.0/22 for some configurations). If no rule admits those sources on the health-check port, the backend is marked unhealthy while the app answers every internal request.

Work the diagnosis in this order:

1. Confirm the health check protocol, port, and path. A check on /healthz:8080 against an app listening on 443 fails regardless of firewall state.
2. Check whether the allow rule for the prober ranges still targets the instances. The release may have changed the instance template's network tags, so a tag-targeted rule silently stopped matching. gcloud compute firewall-rules list --filter="sourceRanges:35.191.0.0/16" shows what exists, but you must compare against the tags actually present on the new instances.
3. Check priority shadowing. With 140 hand-written rules, a deny rule with a lower priority number than the prober allow rule matches first and the allow is never evaluated. gcloud compute firewall-rules list --sort-by=priority exposes this.
4. Confirm the app binds 0.0.0.0, not 127.0.0.1. A curl run on the VM against localhost succeeds even when the app is unreachable on the VM's actual address.

Fix: one rule, target = the app tier's service account, source = 35.191.0.0/16 and 130.108.0.0/16, protocol/port = the health-check port, priority above any competing deny. Note that a passthrough load balancer does not require a proxy-only subnet; that requirement belongs to internal Application Load Balancers and proxy Network Load Balancers, so do not go looking for a proxy-only subnet as the cause.

Most likely cause: IP forwarding is disabled on the NVA, so Google Cloud drops every packet it tries to forward.

When a route uses --next-hop-instance, the VM receives packets whose source address belongs to someone else. A VM can only emit packets with a foreign source IP if canIpForward is true. With it false, the packet is discarded before it leaves the appliance's NIC - which matches both symptoms exactly: total egress loss for the clients and zero captures on the appliance.

Checklist for an NVA next hop:

1. canIpForward enabled on the instance. gcloud compute instances describe nva --format="value(canIpForward)" to verify; gcloud compute instances update nva --can-ip-forward to fix.
2. Route priority below 1000. The VPC's default route to the internet gateway has priority 1000; a custom 0.0.0.0/0 route only wins if its priority is lower (for example 800).
3. Ingress firewall rules on the NVA permitting the client subnets, and egress rules permitting the NVA to reach the internet.
4. Internet access for the NVA itself. The NVA's own egress cannot come back through the clients; it needs Cloud NAT on the NVA's subnet or an external IP.
5. Return path. Replies from the NVA to the clients are routed by destination, so they go straight to the client subnet; that asymmetry is normal in Google Cloud. Ensure no deny rule blocks 10.0.0.0/8 into the client subnets.
6. MTU alignment. If the NVA or its tunnels use 1460, set the VPC MTU accordingly or large packets will black-hole.

Operational advice: a single NVA is a single point of failure for all egress. Put two appliances behind an internal passthrough load balancer and point the route at the load balancer, or use policy-based routing to scope the inspection to the subnets that actually need it rather than 0.0.0.0/0.

No. The expansion is impossible as stated, because any /20 containing 10.0.0.0 also contains the app and database subnets.

gcloud compute networks subnets expand-ip-range only allows a new prefix that is a supernet of the existing range and that does not overlap any other subnet in the VPC. 10.0.0.0/20 spans 10.0.0.0 through 10.0.15.255, which swallows 10.0.1.0/24 and 10.0.2.0/24. Even a /23 (10.0.0.0/23) collides with 10.0.1.0/24. There is no legal expansion in this layout, so the request is a renumbering project, not a change ticket.

Zero-downtime plan:

1. Create a new subnet, for example 10.0.3.0/22, in the same region and VPC. /22 gives 1,022 addresses, roughly five times current headroom.
2. If the web tier sits behind a managed instance group and the load balancer, roll the instance template to specify the new subnet and perform a rolling replace. The load balancer drains each instance; no client sees downtime and no surviving instance is re-addressed.
3. Keep the old subnet in place until the last old instance is gone, then delete it. Update firewall rules, health checks, and any address-based references before removal; prefer service-account or tag targeting so the migration does not require touching 140 rules.
4. If any instance genuinely must keep its address (stateful, or an external allowlist peer), that instance requires its own migration plan - address reservations do not survive a subnet change for VMs, so plan a DNS or load-balancer cutover rather than an IP-preserving move.

Do not schedule the in-place expansion: it will be rejected and the attempt signals to the team that the layout is not understood.