Stateful Kubernetes Workloads with Local Volumes and Aggressive Scale-In

A PodDisruptionBudget is a pod-count control, not a data-plane safety control. maxUnavailable=1 limits voluntary disruptions at the API level, but it does not understand Kafka partitions or under-replication. If two brokers are on the same node, a single node drain is one failure-domain event that removes two brokers. If the first broker's readiness probe reports Ready before Kafka has fully re-replicated its partitions, the eviction API can allow the second eviction. The PDB sees a Ready pod, not a healthy replication factor.

Local persistent volumes bind each pod to a node through node affinity. Once a broker is evicted, it cannot be rescheduled onto a different node; it stays Pending until the original node returns or someone manually intervenes. During scale-in the node may be terminated, so the local data is lost unless Kafka itself replicates the partition elsewhere. That turns a cost-saving scale-in into a data-availability incident: partitions remain under-replicated longer, producer acknowledgements time out, and the broker count is effectively reduced.

The same problem applies to Elasticsearch shards and PostgreSQL replicas on local storage. The scheduler treats the pod as the unit; Kafka treats the partition replica as the unit. The controls must be aligned.

Node pools: separate system, stateless and stateful pools. Taint the system pool CriticalAddonsOnly. Enable cluster autoscaler on the stateless pool. Give stateful workloads dedicated node pools per AZ with scale-in disabled, or use a node group marked not safe to evict. For Kafka, use dedicated nodes with local NVMe and at most one broker per node. For PostgreSQL, prefer network-attached storage with synchronous replicas rather than local PV unless the operator handles replica rebuild.

Scheduling: apply required pod anti-affinity on hostname for Kafka brokers, and preferred or required topology spread across AZs. Set a readiness gate or sidecar probe that checks under-replicated partitions, not just process liveness. Add a PDB of maxUnavailable=1 only as a second layer; the first layer is one broker per node and per failure domain.

Upgrade and drain: never drain a local-PV node unless the workload has a verified replica elsewhere. Use eviction, not --force or --disable-eviction. Upgrade one broker at a time; after each, wait until under-replicated partitions return to zero before proceeding. Reassign partitions before removing a node. For PostgreSQL, fail over and verify replication lag before touching the next replica. Define a maintenance window and an abort condition.

For stateless services, set CPU and memory requests on every pod; without requests the scheduler cannot reserve capacity and the HPA has no baseline. Use HPA on CPU plus a custom metric such as requests per second or queue depth. Add cluster autoscaler overprovisioning with low-priority pause pods so scale-out has immediate capacity. Set scale-up stabilization to zero and scale-down stabilization to several minutes to avoid flapping. Cap the pool maximum and alert when Pending pods exceed a threshold. Do not let cost targets override Pending-pod SLOs.

For stateful services, do not use HPA to add Kafka brokers or Elasticsearch nodes. Scale by adding a broker and reassigning partitions, or by adding a data node and rebalancing shards, with verification. Use vertical sizing or dedicated node pools. Disable scale-in for nodes with local PVs, or require a drain workflow that first moves replicas. For cost reduction, scale stateless pools aggressively, use scheduled scaling for predictable peaks, and use reserved capacity for stateful pools. Measure cost per request and cost per replicated partition, not just node count.