CKAD: Restart Loop on a Slow-Warming Worker with No Shell

Restart Loop on a Slow-Warming Worker with No Shell

Pod ledger-worker in namespace prod restarts roughly every 90 seconds; RESTARTS is 7 after 11 minutes. The app loads a 4 GB ledger snapshot and serves health on :8080 only after the load finishes, which takes about 70 seconds on a cold cache. Its liveness probe is httpGet /livez with initialDelaySeconds: 30, periodSeconds: 10, timeoutSeconds: 1, failureThreshold: 3. The container runs with a read-only root filesystem, the image contains no shell and no tar, and the application writes its real log to /var/log/ledger/app.log inside the container rather than to stdout. kubectl logs shows only the startup banner.

1. Explain the restart loop with the probe arithmetic, and change the minimum number of fields to stop it.

Read the worked answer

Arithmetic: the liveness probe starts at 30s, probes at 30/40/50s, and the third failure at ~50s kills the container - the app needs ~70s before /livez answers at all. So the container is killed roughly 20s before it could ever become healthy, the 4 GB snapshot starts loading again from scratch, and the cycle is self-sustaining. The ~90s period is 50s of warm-up + termination grace + restart delay, which is the signature of a liveness kill rather than a crash (state.lastState.terminated.reason will be Error with exit code 137, not OOMKilled).

Note also timeoutSeconds: 1: if /livez does any work while the snapshot is being written, a 1s timeout turns a slow answer into a kill.

Minimum change - add a startupProbe and leave liveness alone:

startupProbe:
  httpGet: {path: /livez, port: 8080}
  periodSeconds: 10
  failureThreshold: 12       # 120s budget vs ~70s worst case
  timeoutSeconds: 3
livenessProbe:
  httpGet: {path: /livez, port: 8080}
  periodSeconds: 10
  timeoutSeconds: 3
  failureThreshold: 3
readinessProbe:
  httpGet: {path: /livez, port: 8080}
  periodSeconds: 5
  failureThreshold: 2

While the startup probe is unsatisfied, liveness and readiness are suppressed, so the 70s load is invisible to them. The alternative - raising initialDelaySeconds to 90 - works but penalises every later restart of a warm container by 90s of unprotected runtime. Keep liveness aggressive (it is what recovers a genuinely wedged worker) and let the startup probe absorb the cold-start cost.

2. You need the contents of `/var/log/ledger/app.log`. No shell, no `tar`, read-only root filesystem. Give two workable approaches and their tradeoffs.

Read the worked answer

First, establish where the file lives - the read-only rootfs makes this decisive:

kubectl get pod ledger-worker -n prod -o jsonpath='{range .spec.volumes[*]}{.name}{" "}{.emptyDir}{" "}{.persistentVolumeClaim.claimName}{"\n"}{end}'
kubectl get pod ledger-worker -n prod -o jsonpath='{.spec.containers[0].volumeMounts}'

With readOnlyRootFilesystem: true the app cannot be writing to the image layer, so /var/log/ledger must be a mounted volume (emptyDir or PVC). If it is not in volumeMounts, then the writes are silently failing and the log is empty - which is itself the answer.

Approach A - ephemeral container sharing the volume:

kubectl debug -it ledger-worker -n prod --image=busybox:1.36 \
  --target=ledger-worker -- cat /var/log/ledger/app.log

--target gives the ephemeral container a shared process namespace so you can see the app's PIDs. To read the file you must add a volumeMount for the same volume - kubectl debug does not inherit them, so patch the pod's ephemeralContainers directly:

kubectl proxy &
curl -XPOST localhost:8001/api/v1/namespaces/prod/pods/ledger-worker/ephemeralcontainers \
  -H 'Content-Type: application/json' -d @debugger.json

Tradeoff: ephemeral containers cannot be removed once added, and a Debug pod using --target shares the process namespace, letting you kill the app by accident. Also, an emptyDir is per-Pod, so a --copy-to Pod gets a different, empty volume - copying the Pod does not copy the data.

Approach B - kubectl debug --copy-to and mount the volume there is only useful when the volume is a PVC (RWX) or hostPath, since --copy-to creates a new Pod. For an emptyDir it is useless.

What does not work here: kubectl cp needs tar in the target container, and kubectl exec needs a shell or at least a binary; neither is present.

Best long-term fix: make the app log to stdout/stderr, or add a sidecar tailing the file. Reading logs through a debugger is a one-off; a pod that only exposes logs via a file is unobservable by design.

3. Something exits with code 137. How do you tell a liveness kill from an OOMKill, and why does it matter for the fix?

Read the worked answer

Exit code 137 means SIGKILL, which both the kubelet (liveness failure) and the kernel OOM killer produce. The distinguishing field is lastState.terminated.reason:

kubectl get pod ledger-worker -n prod -o jsonpath=\
'{.status.containerStatuses[0].lastState.terminated.reason}{" "}{.status.containerStatuses[0].lastState.terminated.exitCode}{"\n"}'
# OOMKilled 137   -> kernel killed it, raise limits.memory
# Error     137   -> kubelet killed it after liveness failures

Corroborate:

kubectl describe pod shows Last State: Terminated, Reason: OOMKilled and often a Memory cgroup out of memory kernel event; a liveness kill instead logs Liveness probe failed: events preceding the restart, and the Started/restart timestamps sit ~50s apart.
kubectl top pod (if metrics-server is present) shows RSS near limits.memory before the restart for OOM; near-zero for a liveness kill.

Why it matters: the exit code alone is not actionable. If this is an OOMKill, adding a startupProbe changes nothing - the container still dies at the moment the 4 GB snapshot is paged in, and raising limits.memory is the fix (with a Pod QoS reclassification to Guaranteed if requests are also raised). If it is a liveness kill, raising memory wastes node capacity and hides the real bug. Getting this backwards is the most common wasted day on a "CrashLoopBackOff" ticket, and a 4 GB ledger load is exactly the workload where both look identical from kubectl get pods.