Pod IP allocation failures after node scale-out
A 260-node Kubernetes cluster runs Cilium 1.15 with cluster-pool IPAM. The pod CIDR is 10.244.0.0/16 and each node receives a /24, so the cluster has exhausted all 256 available per-node blocks. The team adds 20 nodes to handle a seasonal batch workload. New pods on those nodes stay in ContainerCreating, and cilium-agent logs on the new nodes report 'failed to allocate IP: no more IPs available in allocation pool'. Existing nodes run only 30-60 pods each, so per-node address space is not the immediate bottleneck. The platform team cannot take the whole cluster down, and the on-prem routers already carry a summary route for 10.244.0.0/16. They propose either reducing the per-node mask to /25 or adding a second pod CIDR. A second proposal is to switch to Kubernetes IPAM with a larger node CIDR mask.
1. Explain why new nodes fail while existing nodes continue to run pods, and identify the first three commands or checks you would use to confirm the diagnosis.
Read the worked answer
Cilium cluster-pool IPAM hands out per-node blocks from the configured cluster CIDR. With 10.244.0.0/16 and a /24 mask, the allocator can create exactly 256 node blocks. When the 257th node asks for a block, the request fails even though individual nodes have free IPs. Existing pods keep their addresses because they already hold blocks; the failure is at block allocation, not per-pod allocation.
First checks:
kubectl get ciliumnode <new-node> -o jsonpath='{.spec.ipam.podCIDRs}'should be empty on a failing node and populated on a working node.kubectl logs -n kube-system <cilium-agent-pod> -c cilium-agent | grep -i 'allocate\|ipam'to confirm the pool exhaustion message.cilium-dbg status --verboseinside an agent pod, pluskubectl get ciliumnodes -o custom-columns=NAME:.metadata.name,PODCIDR:.spec.ipam.podCIDRsto see how many blocks are allocated.
This distinguishes cluster-pool exhaustion from CNI plugin failure, kubelet misconfiguration, or a per-node /24 limit.
2. Evaluate the two proposals: /25 per node versus adding a second cluster-pool CIDR. Which is safer given the constraints, and what are the trade-offs?
Read the worked answer
The /25 proposal is the weaker option. Reducing the mask size does not create free space if every /24 in 10.244.0.0/16 is already allocated; the allocator cannot split an existing node block that another node owns. It also cuts each node to 126 usable pod IPs, which may be tight for the batch nodes. Changing mask size on a live cluster should be treated as a re-IPAM event, not a quick fix.
Adding a second cluster-pool CIDR, for example 10.245.0.0/16, gives new blocks without touching existing pods. The trade-off is routing: the on-prem routers only summarize 10.244.0.0/16, so 10.245.0.0/16 must be advertised or routed before new pods use it. The safer path is to add the second CIDR, update routing, let new nodes draw from it, and only then rebalance or recycle old nodes. Migrating to Kubernetes IPAM with a larger node mask is also valid but requires node and pod churn and careful kube-controller-manager changes, so it is a larger maintenance window.
3. Outline a safe expansion sequence that avoids pod restarts for existing workloads and keeps the on-prem route working.
Read the worked answer
Sequence:
- Add
10.245.0.0/16toclusterPoolIPv4PodCIDRListwhile keeping10.244.0.0/16; do not remove the old CIDR. - Update the on-prem router or BGP advertisement to reach
10.245.0.0/16before scheduling pods that need it. - Roll the Cilium agents, or apply the Helm change and let the DaemonSet roll, so the operator and agents see the new pool. Existing pods are not restarted by an agent roll.
- Cordon and drain or recycle only a small number of old nodes if block rebalancing is needed; new nodes should receive blocks from the new CIDR.
- Verify with
kubectl get ciliumnodethat new nodes have10.245.x.y/24, then run a pod-to-pod and pod-to-on-prem connectivity test.
Do not remove 10.244.0.0/16 until no CiliumNode or pod still references it.