Liveness Probe Kills a Cold-Starting Service

The readiness probe worked exactly as designed and that is precisely why it did not help.

1. New pods start. Readiness (/healthz 503 for 45 s) keeps them out of the Service endpoints, so they receive no traffic. Good so far.
2. The liveness probe budget is initialDelaySeconds 8 + failureThreshold 3 x periodSeconds 5 = 23 s. At ~23 s the container is still warming up, liveness fails three times, and the kubelet sends SIGTERM, then SIGKILL after the grace period.
3. The restart resets the warm-up clock to zero, so the pod can never reach 45 s. It loops forever in CrashLoopBackOff.
4. Because the rollout used maxUnavailable: 25% and maxSurge: 25%, the controller scaled the old ReplicaSet down as new pods were created; the old pods were gone before the new ones proved healthy. There is no surviving capacity to serve traffic.

The key judgement: readiness governs traffic, liveness governs restarts. A slow-but-healthy pod is a readiness problem, not a liveness problem. Liveness should only ever fire when the process is genuinely wedged - a condition a restart can fix.

Add a startup probe on the same endpoint and leave the liveness/readiness thresholds alone. The startup probe gates the other two: liveness and readiness do not evaluate until startup succeeds, so a slow first boot is tolerated without weakening steady-state failure detection.

startupProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 18      # 18 x 5 = 90 s budget vs ~45 s warm-up
  timeoutSeconds: 2
livenessProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 3       # ~15 s to detect a truly wedged process
  timeoutSeconds: 2

Two supporting changes worth making in the same PR:

• maxUnavailable: 0 with maxSurge: 1 (or 25%) so old pods are only terminated once new ones are Available. Slower rollout, no cliff edge.
• timeoutSeconds: 1 was too tight for a JIT-warmed process under load; 2 s avoids probe-induced false positives at high CPU.

Size the startup budget from the p99 cold-start time, not the median, and add a little headroom - 90 s against a 45 s observed warm-up is reasonable.

It does stop the crash loop, and it is a one-line change, but it trades one failure mode for another.

• Slower fault detection, permanently. The 60 s delay applies to every container start, including restarts after an OOMKill or a node failure. A genuinely deadlocked process now goes undetected for a minute instead of ~15 s, which at 8 replicas can mean a long tail of hung requests.
• It is a guess that drifts. The number is coupled to cache size and startup code. The next release that pushes warm-up to 70 s reintroduces the outage silently, and nobody remembers why 60 was chosen.
• A startup probe expresses the intent. It says "this process is allowed a long first boot, and once it is up, judge it strictly." The liveness thresholds stay tight and meaningful.

There is one case where initialDelaySeconds is acceptable: a probe that is cheap and where the delay is genuinely fixed, e.g. a sidecar with a deterministic 2 s bind. For a variable, data-dependent warm-up, use the startup probe. Note also that neither option fixes the rollout strategy - without maxUnavailable: 0 you can still lose all capacity to a bad image.