Pod CIDR Exhaustion on a VPC-Native Cluster
A production Kubernetes cluster runs VPC-native (alias IP) networking. The cluster's pod secondary range is 10.16.0.0/21, and every node is allocated a /24 from it. The node pool autoscales from 3 to 40 nodes, and the platform team plans to run 40 nodes at the current pod density. On the ninth node the managed control plane refuses to create it, reporting that there is no free IP space in the pod range. Separately, a new data pipeline must reach an on-premises network that has already been allocated 10.16.0.0/16 over a VPC peering, and a staging cluster for the same product must live in the same project with its own non-overlapping ranges. Nobody wants to reduce max-pods below 110 per node, because the workload is many small sidecars rather than few large pods.
1. Show the arithmetic behind the failure and identify precisely which design decision is wrong: the `/21`, the `/24` per node, or the 40-node target.
Read the worked answer
Pod-range capacity is 2^(32 - prefix). A /21 offers 2,048 addresses. With a /24 per node (256 addresses, minus a handful of provider-reserved addresses per range) only 8 nodes can ever be created, regardless of autoscaler settings. The failure on node 9 is exactly the predicted ceiling, not flakiness.
So the /24-per-node choice is compatible with the goal (/24 leaves roughly 250 usable addresses, comfortably above max-pods: 110), and the 40-node target is a business requirement. The defect is the size of the parent pod range: a /21 cannot host 40 children. Forty nodes need at least 40 x 256 = 10,240 addresses, so a /18 (16,384) is the minimum sensible parent with headroom for surge nodes and replacement nodes during upgrades.
The useful framing for the exam is that these three numbers are coupled: parent range size, per-node range size, and node count. Changing any one silently constrains the other two, and the constraint surfaces only at scale, weeks after the design review.
2. Propose an address plan for production, staging, and the on-premises peering. Include the migration approach, given the existing pod range cannot be resized in place.
Read the worked answer
Start from the collision: on-premises already claims 10.16.0.0/16, which fully contains the current production pod range 10.16.0.0/21. Peering routes will either be rejected or, worse, remain ambiguous. Production pods must therefore move out of 10.16.0.0/16 entirely, not just out of the /21.
A workable layout:
- Production pods:
10.32.0.0/18(16,384 addresses, 64 nodes at/24), services10.36.0.0/20. - Production nodes: a separate smaller range from
10.32.64.0/20. - Staging pods:
10.40.0.0/18, no overlap with production or on-premises. - On-premises peering:
10.16.0.0/16, untouched.
Migration reality: secondary ranges on existing node pools generally cannot be resized. Do not attempt an in-place edit. Create a new node pool or new cluster with the new range, run both, and shift traffic with weighted DNS or a load balancer while draining the old pool. Validate that every consumer that whitelists pod IPs -- firewalls, database grants, legacy services -- has been updated before the cutover, because pod IPs are changing regardless of whether the pods themselves changed.
3. What is the real cost of shrinking the per-node range to `/26` instead, and what else must change with it?
Read the worked answer
A /26 gives 64 addresses per node, and the provider reserves a few of them, so usable pod addresses land around 58-60. max-pods on the node must be set below that usable number -- around 55 -- or the kubelet will accept pods it cannot address, which manifests as pods stuck in ContainerCreating with sandbox or CNI errors rather than a clean scheduling refusal.
Given the 40-node target and the sidecar-heavy workload, halving density roughly doubles the node count needed for the same pod count, which doubles node cost, kube-proxy and CNI state, and the blast radius per node failure. That is usually the wrong trade when the real problem is a parent range that is one or two bits too small.
The legitimate case for /26 is when the cluster must fit inside an address plan you do not control -- for example, a shared corporate 10.x with no large free block, or a cluster that peers with many external networks. Then the constraint is external and density is the price. The decision rule: size the parent range to the node ceiling you actually plan to reach, do not shrink the child range to fit a parent chosen arbitrarily.