Graceful Shutdown and a Disruption Budget That Blocks a Drain

The failure is a race between endpoint removal and signal delivery, and it happens on every rolling update:

1. The pod is marked for deletion. The kubelet sends SIGTERM at the same moment the pod enters Terminating.
2. Endpoint removal is propagated asynchronously: the API server updates the Endpoints/EndpointSlice, then kube-proxy rules on every node and the ingress controller's backend list are updated. This takes hundreds of milliseconds to seconds.
3. In that window the pod is still in the load-balancer rotation but has already stopped accepting connections. Requests routed to it are refused → 502/503.

terminationGracePeriodSeconds: 30 is not a delay before SIGTERM; it is the upper bound between SIGTERM and SIGKILL. Since the app exits after ~1 s, 29 s of the budget go unused and the pod disappears while still registered.

The fix is a preStop sleep that keeps the process serving while the endpoints converge:

lifecycle:
  preStop:
    exec: { command: ["sleep", "10"] }
terminationGracePeriodSeconds: 40

Order the budget so grace period > preStop + worst-case request time (9 s), and have the app stop accepting new connections on SIGTERM while finishing in-flight ones. Do not rely on a preStop HTTP call to a health endpoint alone - the sleep also covers the case where endpoint propagation outlasts the deregistration.

Two constraints compound:

• PDB math: minAvailable: 11 with 12 replicas permits exactly one voluntary disruption at a time. kubectl drain uses the Eviction API, which returns 429 for every pod that would breach the budget, retries with backoff, and blocks. On a six-node cluster the node may hold 2-3 replicas, and each eviction must wait for its replacement to become Available before the next is permitted. That serialization is the 25 minutes.
• No headroom to replace them: the drain cordons the node before evicting, and with the autoscaler disabled there is nowhere for the replacement pods to go. Requests bounce between Pending because the remaining five nodes have no free requested capacity - again, top may well show free RAM that the scheduler cannot use.

Changes, in order of effect:

1. Provide capacity or a scale-out path. Enable the autoscaler, or pre-scale before maintenance. Without it, any PDB that blocks is fatal rather than merely slow.
2. Express the budget as a fraction with slack. minAvailable: 11 on 12 replicas is a 92% availability floor for a rollout; consider maxUnavailable: 2 (or 25%) if the SLO tolerates it, and keep the strict PDB only for true singleton services.
3. Decouple maintenance from serving. maxUnavailable: 0 on the Deployment is right for user traffic but means every eviction must fully replace a pod. For maintenance windows, run drains during low traffic and give them an explicit --timeout so they fail loudly instead of hanging.

Note the interaction to call out: maxUnavailable: 0 plus a near-saturated PDB means no pod is allowed to be down, which is a policy contradiction, not a tuning problem.

No. The 502s are caused by the pod leaving the request path before routing rules are updated. Grace period controls only how long the kubelet waits between SIGTERM and SIGKILL. If the process exits in ~1 s, raising the grace period from 30 to 60 changes nothing - the SIGKILL is never reached. The signal timing, not the deadline, is the bug.

The only ways to keep serving during endpoint convergence are to delay shutdown (a preStop hook) or to make the app drain in-flight requests and keep the listener open for a bounded period after SIGTERM. Both need the grace period to be long enough, so a larger value is a necessary but not sufficient condition - it must exceed preStop + drain, which 30 s already does for a 10 s hook plus 9 s requests.

Grace period would matter if the app drained for 35 s; then 30 s would SIGKILL mid-drain and you would see truncated responses as well as refused connections. Worth checking the application's shutdown logs for SIGKILL-shaped deaths before concluding.

Metric: the rolling 5xx ratio at the ingress during a deploy - specifically the count of 502/503 originating from upstream connection resets, not total errors. Watch it across a canary rollout; it should fall to the background rate (ideally zero) rather than the ~40-per-pod burst. Pod-side, container_last_seen gaps relative to kube_pod_deletion_timestamp are a useful secondary signal.