OOMKills and Probe Starvation on Kubernetes at Orbit Analytics

These are two independent failure modes that reinforce each other.

OOMKills (exit 137):

• -Xmx900m is heap only. The container's total footprint is heap plus metaspace, thread stacks, code cache, direct byte buffers, and JVM overhead - comfortably above the 1Gi limit. The kernel kills the process before the JVM throws OutOfMemoryError, which is why there is no Java heap dump.
• Fix: bring the heap into a defensible ratio with the limit. Either raise the memory limit (for example 1.5Gi) and keep -Xmx sized with headroom, or set the heap as a percentage of the limit (-XX:MaxRAMPercentage) so it cannot drift. Then load-test at spike volume and watch working-set memory against the limit.
• Do not "fix" this by only raising the limit without checking for a leak: OOMKill every six hours under a daily spike could also be steady growth. Take heap histograms across cycles.

Probe starvation:

• Liveness and readiness share /health, which does a database query on the same thread pool as business traffic. Under load, requests queue, the probe times out, and the kubelet restarts a pod that was serving fine - a death spiral, because restarting throws away the warm cache.
• Fix: liveness must be cheap and local - process liveness only, no dependency calls. Readiness may check dependencies and should reflect ability to serve. Add a startupProbe with a generous failure threshold to cover the 4-minute warm-up so liveness does not kill pods during initialization. Set realistic timeouts and failureThreshold based on measured p99 under load.

Sequence: probes first (cheap, immediate reduction in self-inflicted restarts), then memory sizing with load evidence.

Why it fails: the HPA compares observed CPU against the request, not the limit. Observed CPU near 30% against a 250m request means the 70% target is never crossed. That does not mean the service is idle - it means the request is a poor model of actual usage. Requests of 250m may have been set by guess and never revisited.

Second problem: CPU may not be the right signal at all. If the user-visible bottleneck is memory pressure and a 4-minute cache warm-up, then CPU utilization is a lagging, misleading proxy. Scaling out adds pods that each take four minutes to become useful, which is why adding replicas "did not help much."

Changes:

1. Right-size requests from observed data, e.g., p95 usage over a full week including the 09:00 spike. Lower requests make the HPA target meaningful; do it alongside the memory limit change so the two are coherent.
2. Consider a better scaling signal: requests per second per pod, queue depth, or latency-driven scaling, via custom or external metrics. Scale on the thing that actually degrades during the spike.
3. Address warm-up directly: pre-warm the cache in an init container or startup phase, share a distributed cache, or pre-scale on a schedule ahead of the known 09:00 spike. A predictable spike is the easiest autoscaling case - a cron-based minimum replica bump before 09:00 plus HPA for headroom.
4. Stabilization windows so the HPA does not flap, and behavior.scaleUp policies tuned to how fast a pod actually becomes ready.

Autoscaling cannot compensate for a slow-start pod and a mis-sized request at the same time; fix both.

minAvailable: 6 with exactly 6 replicas means zero voluntary disruptions are permitted. Every node drain blocks indefinitely, so the upgrade either stalls or someone deletes the PDB under pressure - the worst possible process.

PDB redesign:

• Set minAvailable: 5 (or maxUnavailable: 1) with 6 replicas, allowing one pod to be evicted at a time while preserving capacity for the loss of a single pod.
• Ensure the workload has headroom: run 7-8 replicas during maintenance windows so a PDB of maxUnavailable: 1 does not reduce serving capacity below peak need. Pair with topology spread constraints so replicas sit across zones and nodes.
• Verify the readiness probe is meaningful, because the eviction-replacement cycle depends on it. With the current shared /health probe and 4-minute warm-up, a drained pod's replacement may be judged ready too early or restarted mid-warm-up - fix the probes first.

Rollouts:

• Use maxSurge: 1, maxUnavailable: 0 with a readiness gate and a startupProbe, so new pods must be genuinely ready before old ones retire.
• Add a preStop hook and adequate terminationGracePeriodSeconds so in-flight requests drain rather than error.
• For higher assurance, canary a subset via a service mesh with automated rollback on error rate.

Upgrade practice: cordon and drain one node at a time with a timeout, monitor error budgets during the window, and rehearse in a staging cluster with production-like replica counts. A 5-hour upgrade is a symptom of a PDB that permits nothing and probes that cannot be trusted.