Reconciliation Pod with a Fetch-Before-Start Dependency

Three containers, two volumes.

spec:
  volumes:
  - name: work      # shared output
    emptyDir: {}
  - name: manifest  # fetched input, read-only to consumers
    emptyDir: {}
  initContainers:
  - name: fetch
    image: curlimages/curl:8.7.1
    command: ['sh','-c','curl -fsS --retry 5 --retry-connrefused -o /manifest/manifest.json http://intake.internal:8080/manifest']
    volumeMounts: [{name: manifest, mountPath: /manifest}]
  containers:
  - name: main
    image: registry.local/recon:1.4
    volumeMounts:
    - {name: manifest, mountPath: /etc/recon, readOnly: true}
    - {name: work, mountPath: /out}
  - name: uploader
    image: registry.local/uploader:0.9
    volumeMounts: [{name: work, mountPath: /out}]

fetch must be an initContainer: its completion is what gates startup of main, so you get ordering for free and no application code has to poll for the manifest. It should not be a sidecar, because nothing later in the Pod depends on it being alive.

uploader must be a regular container, not an initContainer: it has to run concurrently with main, watching files as they appear. Writing it as an init container would deadlock the Pod.

/out is an emptyDir because both main and uploader need concurrent read/write access on the same node, and the data is regenerated nightly. manifest is a separate emptyDir (not a ConfigMap) because it is fetched at runtime and is far too large for the 1 MiB object limit; mounting it read-only into main documents that main never rewrites its input.

Set sizeLimit: 1Gi on work so a runaway run is evicted rather than filling the node's 2 GiB, and give the uploader its own resources so it is not starved by main.

What happens: fetch exits non-zero. The Pod stays in phase Pending with STATUS: Init:0/1 forever from the user's point of view; the kubelet retries the init container with exponential backoff (10s, 20s, 40s, ... capped at 5 minutes). restartPolicy: Always (the only policy for Pods, and the default) applies, so it retries indefinitely rather than failing the Pod. No main container is ever created, so there is no output to lose.

Observation:

kubectl get pod recon                       # Init:0/1, RESTARTS 0
kubectl describe pod recon | tail -20       # 'Back-off restarting failed container fetch'
kubectl logs recon -c fetch --previous      # curl: (7) Failed to connect
kubectl get pod recon -o jsonpath='{range .status.initContainerStatuses[*]}{.name}={.state.waiting.reason}{"\n"}{end}'

The initContainerStatuses[].state.waiting.reason is the field that tells operators which stage is failing (CrashLoopBackOff on fetch reads very differently from one on main). Keep the --retry --retry-connrefused flags inside the command so a 60s outage is usually absorbed within a single attempt and never shows up as a backoff event; the backoff is the coarse safety net, not the primary retry mechanism.

Bound it: add activeDeadlineSeconds: 3600 to the Pod so a permanently broken intake does not leave an Init pod around forever.

Job instead? Only if you want Pod-level retry accounting (backoffLimit, ttlSecondsAfterFinished) and completion semantics. A Job still cannot start main before the fetch finishes without an init container, and a Job whose Pod runs a long-lived uploader never reaches Complete. You would also lose the shared emptyDir between fetch and uploader. Not worth it here.

Container restart: files survive. An emptyDir is scoped to the Pod, not the container. The kubelet restarts only the uploader container inside the same Pod sandbox, so the volume (a directory on the node's disk) is untouched and main keeps writing to it. You can confirm which container restarted:

kubectl get pod recon -o jsonpath='{.status.containerStatuses[?(@.name=="uploader")].restartCount}{"\n"}'
kubectl get pod recon -o jsonpath='{.status.containerStatuses[?(@.name=="uploader")].lastState.terminated.reason}'   # OOMKilled

If lastState.terminated.reason is OOMKilled, the fix is a higher resources.limits.memory, not a different volume: the Pod-level limit kill also takes down main.

Node drain / eviction / reschedule: files are lost. Deleting the Pod deletes the emptyDir, and the replacement Pod lands wherever the scheduler puts it with an empty directory. Also note sizeLimit eviction: if work exceeds its limit the Pod is evicted, not just the volume trimmed.

If the output must survive a drain, switch work to a ReadWriteOnce PVC mounted at /out in both containers. Constraints to check first: RWO allows one node at a time, which is fine while both containers are in one Pod, but a replacement Pod cannot mount it until the old one's detach completes - so a drain has a gap. If the uploader must keep up with files across that gap, use RWX storage or accept that the run restarts from scratch (which is why the nightly run should be idempotent per account).

Compare the two timestamps the API server already records:

kubectl get pod recon -o jsonpath='{.metadata.creationTimestamp}{"\n"}{.status.initContainerStatuses[0].state.running.startedAt}{"\n"}'

• initContainerStatuses[0].state.running present with a recent startedAt = the fetch is executing right now (healthy, just slow).
• state.waiting.reason: CrashLoopBackOff, restartCount climbing, lastState.terminated.exitCode = the fetch is failing and being backed off. The restartCount growth rate tells you whether the endpoint is flapping (slow growth) or the URL/credentials are wrong (fast, immediate failure every time).
• state.terminated.exitCode: 0 but the Pod still Init = the fetcher succeeded and a later init container is the problem.

A useful alert condition is restartCount > 3 within 10m on the fetch container specifically, rather than "Pod not Running", which fires for the healthy slow-fetch case and trains operators to ignore it.