Nightly batch memory pressure evicts playback pods

Build a timeline at 02:00. In the Kubernetes cluster explorer, compare node allocatable memory, requested memory, and actual working set for general and batch nodes. Query pod-level memory used versus requested and limit, grouped by namespace and workload, to see whether batch pods are running on general nodes and bursting beyond their requests. Check eviction counts, OOMKills, pod restarts, and pending pods; connection resets in playback are consistent with eviction, not application exceptions. Use APM entity relationships to map playback pods to nodes and confirm the evicted pods were on the pressured nodes. The key signal is the gap between 38 GB requested and 64 GB allocatable while nodes still hit 90%: requests are understated and batch has no limits, so actual memory is invisible to the scheduler. That points to noisy-neighbor pressure and scheduling, not simple cluster capacity. Adding eight nodes might move the problem, but it does not fix unbounded batch workloads or protect ingestion.

First make batch behavior visible and bounded, then isolate it. Add resource requests and limits to batch pods based on observed working set, using a canary namespace before rolling out. Create a dedicated batch node pool with taints and tolerations or node affinity so transcoding cannot consume general-node memory. Add priority classes so ingestion and playback outrank batch, and PodDisruptionBudgets for stateful ingestion. Use a ResourceQuota and LimitRange per namespace to prevent future unbounded pods. To stay cost-neutral, rebalance the existing 30 nodes rather than adding eight; for example, move general capacity to the batch pool after measuring batch peak. Monitor node memory pressure, eviction rate, OOMKills, pending pods, batch duration, playback p95, and ingestion error rate. Roll out in stages with a rollback if batch jobs fail or ingestion latency rises. The evidence that would change the plan: batch peak exceeds the isolated pool, in which case add capacity only after rightsizing.

Alert on pressure and eviction, not just a high memory percentage. Add a Kubernetes node memory-pressure or eviction-rate condition, a pod restart/OOMKill condition for playback and ingestion, and a pending-pod condition. Keep a static threshold for node memory above 90% for a sustained window, but pair it with a lower-severity condition during the batch window so the team gets early warning. Use a workload health view that groups batch, playback, and ingestion entities. Add a synthetic playback check and a checkout SLO burn-rate alert as customer-facing guardrails. Route batch-capacity warnings to the platform team and customer-impact alerts to the on-call. Review the first three batch runs: if the alert fires after evictions, shorten the window or lower the threshold; if it fires every night without impact, scope it to the batch workload and tune it. Do not mute the node pool globally; suppress only expected batch scale events with tags.