Pending Pods and Image Pulls on Private NKS Nodes
MetroCart runs an NKS cluster in KR1 with three node groups. ng-general has 3 nodes (8 vCPU/32 GB) in a public subnet and cluster autoscaler enabled, min 3, max 6. ng-batch has 2 nodes (16 vCPU/64 GB) in a private subnet with autoscaler disabled. ng-gpu has 1 GPU node in a private subnet. The API deployment has 40 replicas with requests of 1.5 CPU and 3 GiB and a nodeSelector of workload=general. A separate batch deployment has 10 pods with requests of 4 CPU and 16 GiB and a nodeSelector of workload=batch. Many API pods and six batch pods are Pending. The new API image is stored in NHN Container Registry (NCR). API pods on ng-general pull it, but pods scheduled to private node groups fail with ImagePullBackOff. The cluster has no NAT gateway. The team wants to keep private nodes private.
1. Show the capacity math for the Pending API pods, explain why the autoscaler does not fully solve the problem, and state the smallest change that makes all 40 replicas schedulable.
Read the worked answer
Requests are 40 x 1.5 = 60 vCPU and 40 x 3 = 120 GiB. At max 6 nodes, ng-general provides 48 vCPU and 192 GiB, but system pods consume roughly 2 vCPU and 4 GiB, leaving about 46 vCPU. That is still 14 vCPU short, so even a fully scaled group cannot schedule all 40 replicas. The autoscaler is enabled, but it cannot exceed the group max. The smallest change is either raise max to 8 (8 x 8 = 64 vCPU, about 62 usable, which fits 60) or lower per-pod requests to about 1.1 CPU (44 vCPU). Raising max costs two more nodes; lowering requests risks under-provisioning and noisy neighbors. Choose based on measured usage, not just pod count.
2. The batch pods are also Pending. Why does the cluster autoscaler not help, and what are two safe fixes?
Read the worked answer
The batch deployment requests 10 x 4 = 40 vCPU and 160 GiB. ng-batch provides 2 x 16 = 32 vCPU and 128 GiB, minus system overhead, so it is short about 10 vCPU and 32 GiB. The autoscaler cannot help because it is disabled on ng-batch. Safe fix one: enable autoscaler on ng-batch with min 2 and max 3; 3 nodes give 48 vCPU and 192 GiB, enough for 40 vCPU plus system pods. Safe fix two: create a separate batch node group with a higher max and keep the current group fixed, which limits blast radius. Do not schedule 16 GiB batch pods onto 32 GB general nodes; one large pod per node wastes capacity and can evict API pods.
3. Explain the ImagePullBackOff on private node groups and give a fix that preserves the no-public-IP requirement.
Read the worked answer
Private nodes have no default route to the internet, so they cannot reach the public NCR endpoint to pull the image. ng-general nodes are in a public subnet and can reach it. Add a NAT gateway in the public subnet, add a 0.0.0.0/0 route in the private app subnet's route table pointing to the NAT gateway, and allow outbound TCP 443 from the node security group to the NCR endpoint. The nodes still have no public IPs and accept no inbound internet traffic. If NCR private endpoint or VPC endpoint access is available in the region, prefer that and avoid NAT data charges. After the change, delete and recreate the failing pods so kubelet retries the pull.