Moving Bursty Builds to Kubernetes Cloud Agents
Your Jenkins controller runs 40 static agent VMs, each with four executors. At peak the queue waits 20 minutes, yet CPU on the agents averages 20%. Two agents are offline because their workspace disks filled. The team wants to move builds to a Kubernetes cloud using the Kubernetes plugin. The cluster has a 30-pod namespace quota, a mix of on-demand and spot node pools, and a network policy that blocks TCP 50000. Builds need Maven, Docker (for image builds), and a 2 GB test database container. Some pipelines run 45 minutes; nightly jobs run 3 hours. The controller is at https://ci.example.com behind an ingress, and the Kubernetes service account currently has cluster-admin. You must reduce queue time, keep image builds working, and control cost.
1. Pods may stay Pending and agents may never connect. What do you inspect, and how do you fix scheduling and networking?
Read the worked answer
Start with kubectl describe pod and events: Pending usually means insufficient CPU/memory, an unsatisfiable node selector or taint, a PVC that cannot bind, or the 30-pod namespace quota. Check ResourceQuota and LimitRange. If the pod is Running but the agent never connects, the JNLP container cannot reach the controller. The network policy blocks TCP 50000, so either allow egress to the controller's JNLP port or switch the inbound agent to WebSocket over 443 (-webSocket) and expose it through the same ingress. Verify jenkinsUrl and jenkinsTunnel from inside a pod, check DNS, and confirm the service account can create pods. A quick test is to run a debug pod and nc -vz ci.example.com 50000; if that fails, fix the policy or use WebSocket.
2. Design the pod template for Maven, Docker, and the test database. Which fields and resource settings do you choose, and why?
Read the worked answer
Use one pod template with a jnlp container, a maven container, and a docker container or a rootless BuildKit/Kaniko container. Add a test-db sidecar with an emptyDir and a readiness probe. Set resource requests from measurements, for example 500m/1Gi for Maven and 1 CPU/2Gi for Docker and the database; set limits a little higher. Use activeDeadlineSeconds of 3600 for 45-minute jobs and 14400 for nightly, idleMinutes: 5, and podRetention: Never. Mount the Docker socket only on trusted agents; for fork PRs or untrusted code use rootless builds. Give the pod a dedicated service account with only the RBAC it needs. Use labels and nodeSelector to place deploy pods on on-demand nodes and test pods on spot.
3. The static agents are not CPU-bound, yet the queue waits 20 minutes. How do you trade queue time against cloud cost with caps, retention, and node pools?
Read the worked answer
Forty static agents give 160 executors but average 20% CPU, so the queue is not a CPU shortage; it is likely label mismatch, executor slots held by long jobs, or disk-full agents. Kubernetes lets you right-size per build. Set containerCapStr and namespace quota high enough for peak, but cap per cloud to avoid runaway. Use idleMinutes: 5 and podRetention: Never so idle pods do not cost money, and let the cluster autoscaler add spot nodes for test jobs. Keep deploy and nightly jobs on on-demand capacity; spot interruptions should retry at the stage level. activeDeadlineSeconds kills stuck builds and frees quota. Monitor queue wait, pod start latency, and spot interruption rate; if queue wait remains high, raise the cap or quota rather than adding idle static agents.