CKA (CNCF): A ResourceQuota That Stalls a Rollout Every Hour

The quota is exactly saturated before the surge pod is even attempted.

worker:   8 replicas x 750m  = 6000m CPU          8 x 1Gi  = 8Gi
cronjob:  4 pods     x 500m  = 2000m CPU          4 x 2Gi  = 8Gi
total used                   = 8000m = requests.cpu hard limit
                               = 16Gi  = requests.memory hard limit

Both dimensions are pinned at the limit. The ReplicaSet wants to add one surge pod at 750m/1Gi, so the quota admission plugin refuses the create. The error names the first dimension that failed, requests.cpu, but memory would have rejected it too - do not 'fix' only the CPU number.

Why the Job pods count: a ResourceQuota is evaluated over the sum of container requests for every pod in the namespace that is in a non-terminal phase. Running and Pending pods definitely count. Suspended, Succeeded and Failed pods do not. The three-minute jobs are live pods during those three minutes, so they spend the quota the whole time they exist - quota is about concurrent reservation, not about how long the pod lives.

Why it self-heals: once the CronJob's pods reach Succeeded, the quota controller recomputes used and the ReplicaSet's next create attempt is admitted. Nothing retried by hand - the ReplicaSet's controller loop is the retry.

Watch it live:

kubectl describe quota reporting-quota -n reporting   # used vs hard, both dimensions
kubectl get pods -n reporting --field-selector=status.phase=Running

One trap: a pod in Terminating still counts until it is fully deleted, so used can exceed what kubectl get pods appears to show.

Make the rollout reclaim before it allocates: maxSurge: 0, maxUnavailable: 1.

kubectl patch deployment worker --type=merge -p \
  '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":0,"maxUnavailable":1}}}}'

Why it now fits in the same 8-CPU ceiling:

step 1  delete one old pod   -> used = 7 x 750m + 2000m = 7250m   (750m freed)
step 2  create the new pod   -> used = 7250m + 750m     = 8000m   (exactly at the limit)

What it costs: you give up surge headroom. Availability dips to 7/8 for the seconds between the old pod terminating and the new one passing readiness. And because used lands exactly on 8000m, there is zero slack - any other pod appearing in that window (a CronJob starting early, a debug pod, a kubectl run) blocks the rollout again, and if a terminating old pod is still counted when the new pod is admitted, the create is rejected for a few seconds until the goroutine retries.

Better long-term fix: rebalance the requests, not the strategy.

• Drop the CronJob's requests - 4 pods at 500m each for a 3-minute job is usually a request that was never measured. At 250m/1Gi you free 1000m and 4Gi of quota headroom.
• Or raise requests.cpu to 10 and requests.memory to 20Gi, which is honest about peak concurrency.
• Or move the batch work into its own namespace so the two workloads cannot starve each other's rollouts - the standard answer when bursty Jobs and long-lived Deployments share a quota.

Nothing is killed, and the namespace becomes unable to create anything - a much worse state than the one you started in.

What actually occurs when the hard limit is lowered below current usage:

• No eviction, no termination. The quota controller does not touch running pods. It only refuses new creates and updates.
• kubectl describe quota now reads something like requests.cpu 8/6 - used exceeding hard. That is a legal, supported state and it will sit there indefinitely.
• Every pod creation in the namespace is now rejected, not just worker's: the CronJob's next pod, a kubectl run debug pod, a rolling update, and - worst - the replacement for any pod that gets deleted by the ReplicaSet. A single node drain or eviction now takes replicas down with no way to bring them back.
• TLS, kubectl debug, and any operator that creates helper pods in that namespace will also fail.

Undo it in this order:

1. Put the quota back immediately: kubectl patch resourcequota reporting-quota -n reporting --type=merge -p '{"spec":{"hard":{"requests.cpu":"8"}}}'. Creation resumes as soon as used <= hard again.
2. Confirm with kubectl describe quota -n reporting that used is under hard on every dimension, and kubectl get events -n reporting | grep -i quota is quiet.
3. Only then address the original problem - by lowering the workload requests or by right-sizing the quota to observed peak, not by squeezing the ceiling under the floor.

Rule to leave behind: a quota is a ceiling on concurrent requests. Change it based on measured peak usage and a stated headroom, and always check used before lowering hard. kubectl describe quota -o yaml plus kubectl top pods on a representative hour gives you the number; guessing gives you an outage.