ImagePullBackOff 403 and stuck batch pods on a private-node NKS cluster

What the symptom rules out first. Pods that are already running continue to serve traffic, so the image still exists and the tag still resolves. The failure is on the pull path: the kubelet on the node has no usable credential for the registry in that namespace, or the credential it has is no longer authorized. A deleted or wrong tag would produce a manifest-unknown/404 style error, not 403, so do not start by rebuilding the image.

Why it surfaced now. The running pods hold their image layers in the node's local image cache; the kubelet only authenticates when it must actually fetch layers. Rebuilding the tag (or scheduling onto a node that never pulled it) forces a fresh authenticated pull, which is when a missing, expired, or wrongly scoped secret shows up.

Fix.

1. Create a dedicated sub account, for example ci-registry-ro, and grant it only the Container Registry read/viewer permission. Never use the main account key in CI - a leaked main key is effectively full tenancy control.
2. Generate that sub account's own API key pair. NCP Container Registry authenticates with NCP API keys, not a separate registry password, so the secret is a docker-registry type secret built from the access key and secret key.
3. Create the secret in each namespace that pulls (or once per namespace via your manifest tooling) and reference it from the ServiceAccount's imagePullSecrets rather than hand-editing each Deployment, so new workloads inherit it.
4. Roll the Deployment and confirm events show a successful pull. Rotate using two valid keys so pushes never break mid-rotation.

Do not stop here. Also confirm the node actually has a route to the registry endpoint; a credential fix will not help if egress was removed (see the NAT question).

Pending means the scheduler rejected the pod, not that the cluster is slow. Autoscaling is a second-order fix: the cluster autoscaler can only help if a pending pod could actually fit on a new node from that pool. Three things must be checked in order, and they explain the numbers in the setup.

1. Node selectors and taints. The batch pool is tainted. If the batch Deployment has no matching toleration (or has a nodeSelector/label that no longer matches the pool), the scheduler reports 0/13 nodes available: 4 tainted, 9 insufficient cpu and the autoscaler correctly does nothing, because adding nodes to a pool that the pod cannot target changes nothing. High batch-pool CPU around 30% fits this: the pool is not saturated, the pods simply are not allowed onto it.
2. Resource requests versus node size. If each batch pod requests 8 vCPU, or a pod requests more than the largest node type in that pool, no new node can ever satisfy it. Autoscaler max node count in the pool configuration is a hard ceiling too - if the pool is already at maximum, pending pods stay pending forever.
3. Timing. Even a healthy scale-up is minutes: trigger, node provisioning, join, image pull, then scheduling. Fifty minutes suggests repeated failed scale-up attempts or a ceiling, not one slow scale-up.

Changes. Add the correct toleration, set requests from observed usage (not copy-pasted from web tiers), raise the pool's max node count, and add a PriorityClass so interactive workloads win over batch. Separate pools per workload shape, and pre-pull large images so scale-up is not dominated by layer downloads.

The proposal breaks far more than it saves. Worker nodes are in private subnets with no public IP, so their only path to the Container Registry's public API endpoint is the NAT Gateway's egress. Remove it and: new pod schedules cannot pull any image (the 403 incident becomes a total outage), autoscaled nodes come up but never become Ready, and any other outbound dependency - monitoring agent registration, external APIs, OS patching - fails at the same moment.

Routing through the VPN is not equivalent egress. An IPsec VPN to your own data center only reaches your own networks unless the data center hairpins that traffic back out to the internet. Doing so adds latency, consumes the VPN's bandwidth (competing with production admin traffic), introduces asymmetric-routing and NAT-translation problems, and requires firewall and NAT rules on-premises. It also makes the cluster's ability to start pods dependent on a device in your building.

Better options, in order of preference.

• Keep the NAT Gateway but scope it: attach the route only to the subnets that genuinely need outbound access, and keep pools that only consume in-cluster services off that route.
• Reduce the volume, which is what the bill is really charging for. Reuse a small set of base images, pin digests, avoid rebuilding layers, and keep imagePullPolicy: IfNotPresent with pre-pulled images on nodes. NAT Gateway is billed on hourly existence plus data processed, so layer deduplication is the real lever.
• If a private endpoint for the registry inside the VPC is available to you, validate it once, then route pulls privately and drop internet egress for those subnets.

Cost reduction here should come from image hygiene and scoped routing, not from deleting the only egress path.