Cloud NAT Port Starvation on a Private GKE Cluster

The unit of NAT port allocation is the VM - here, the GKE node - not the pod.

With dynamic port allocation disabled, each node receives a static allocation of min_ports_per_vm = 64 ports. Every pod on that node shares those 64 ports after ip-masq rewrites the source address to the node IP. At 110 pods per node, dozens of concurrent TLS handshakes to different destinations exhaust 64 ports almost immediately; new connections fail with endpoint allocation errors, and image pulls time out because the pull is the most connection-hungry thing that happens during scale-out. max_ports_per_vm is effectively unused when dynamic allocation is off, and the two NAT IPs (about 64,512 ports each) are irrelevant because the bottleneck is per-VM allocation.

Changes, in order of impact:

1. Enable dynamic port allocation and raise the floor: min_ports_per_vm=1024, max_ports_per_vm=8192. Nodes then consume ports proportional to real demand instead of a fixed 64.
2. Reduce the number of external pulls. Point the cluster at an Artifact Registry remote repository that proxies the external registry. Pulls then terminate on Google's network, never traverse NAT, and the noisiest consumer disappears.
3. Consider endpoint-independent mapping. It is on by default and consumes a port per destination tuple; turning it off reduces port consumption but breaks some UDP/STUN-style protocols. Keep it on and solve with ports first.
4. Separate noisy neighbours. If a dedicated node pool performs the pulls, give it its own NAT gateway scoped to its subnet so a pull storm cannot starve application traffic.

Verify with Cloud NAT logging enabled: watch for ALLOCATION_ERROR / port-exhaustion events and the port-allocation metrics in Cloud Monitoring, and confirm the errors stop under a scale-out test rather than waiting for the next peak.

Reserve static external IP addresses and attach them to the NAT as the external IP pool.

gcloud compute addresses create meridian-nat-prod-1 --region=europe-west3
gcloud compute addresses create meridian-nat-prod-2 --region=europe-west3
gcloud compute routers nats update meridian-nat --router=meridian-router \
  --region=europe-west3 --nat-external-ip-pool=meridian-nat-prod-1,meridian-nat-prod-2

The reservation is what makes the address stable: an unreserved ephemeral NAT address can change, a reserved address persists until deleted. Give the partner both addresses as /32 allowlist entries and monitor them.

Limits you must be explicit about:

• Cloud NAT is not per-workload. Every VM in the NAT's scope shares the same pool, so one stable IP per environment requires one NAT (and one pool) per environment. The cleanest separation is a separate VPC per environment. If environments share a VPC, create separate NAT gateways scoped to specific subnets with --nat-custom-subnet-ip-ranges and distinct reserved IPs - but you cannot get a distinct IP per pod or per deployment from a single NAT.
• Cloud NAT does not apply to all traffic. Traffic to Google APIs handled by Private Google Access, and traffic to destinations inside the VPC or reachable over Interconnect/VPN, never leaves through the NAT and will not carry the NAT IP. If the partner's API is reachable over an interconnect, the partner sees your private address instead.
• If you need deterministic addressing per workload, terminate egress in an appliance or Secure Web Proxy with a static frontend rather than trying to subdivide Cloud NAT.

Finally, the sporadic 403s are consistent with the partner allowlisting one address while the NAT rotates between two, or with the partner's own WAF reacting to bursty shared-IP traffic. Confirm by correlating the partner's request logs with your NAT IP at the time of the failures.

The absence of NAT flows is expected, not evidence of a fault.

Traffic destined for Google APIs is handled by Private Google Access (or the restricted VIP) and never reaches Cloud NAT, so an empty NAT log is the normal signature of PGA working. The failure lies on a different layer. Work through these in order:

1. DNS. PGA only helps if the name resolves to a Google API VIP: private.googleapis.com (199.36.153.8/30) or restricted.googleapis.com (199.36.153.4/30). If a private Cloud DNS zone for googleapis.com points at restricted.googleapis.com but the subnet is only configured for the default VIPs, connections fail. Run dig +short storage.googleapis.com from inside a pod and compare.
2. Firewall egress. An explicit egress deny or a policy that only allows RFC 1918 destinations blocks 199.36.153.8/30. Egress rules are evaluated top-down by priority like ingress; a hand-written deny from an earlier project phase may be shadowing the allow.
3. Custom routes. If a 0.0.0.0/0 route points at an NVA or a firewall appliance, API traffic follows that route instead of the Google-managed path, and the appliance must forward it correctly. This is the most common reason PGA "stops working" after an inspection appliance is introduced.
4. Kubernetes-level policy. Dataplane V2 enforces NetworkPolicy. An egress deny in the workload's namespace blocks the connection before it ever reaches the VPC, and the VPC sees nothing at all.
5. Workload identity / VPC-SC in a future change. If a VPC Service Controls perimeter is later introduced, the identity calling the API matters as much as the network path.

Best validation: run a Connectivity Test from a node to 199.36.153.8:443 and, separately, curl -sv https://storage.googleapis.com from a pod to see whether the failure is DNS, TCP, or TLS.