Memory-based HPA pinned at maxReplicas starves other Deployments

Mechanically it is correct, and that is the problem. Utilization is mean(working set) / mean(memory request), so 1.65 GiB against a 2 GiB request reads as roughly 82% - above the 70% target - and the controller keeps adding replicas until it hits maxReplicas. Nothing in that loop knows the difference between load and a JVM that simply grew.

Why night changes nothing: heap expands toward -Xmx1400m, metaspace, code cache and thread stacks grow, and the RSS reported by metrics-server includes memory the runtime does not return to the OS promptly. An idle JVM therefore still looks 'full'. Scaling down would require the ratio to fall, which requires the pod to release memory, which it does not.

Raising the target to 85% only moves the threshold; the Deployment would then sit at ~1.7 GiB per pod while claiming to be comfortably provisioned. Adding nodes gets absorbed within days because the request, not the node count, drives the ratio.

Scale on demand instead: CPU with requests that reflect reality, or better a custom/external metric such as requests per second or queue depth through Prometheus Adapter or KEDA. Keep memory as a capacity input - feed VPA recommendations into the request - not as the scaling trigger.

Quantify first: kubectl describe node <n> | grep -A6 'Allocated resources' on all six nodes, plus kubectl get pods -A -o json summed with jq. The answer will be that checkout-api holds 24 × 2 GiB = 48 GiB of requests on a cluster with roughly 180 GiB allocatable. That reservation is the real outage; three extra nodes would be consumed by the same loop.

Order:

1. Stop the bleeding - pin maxReplicas to a number that fits, say 12. The HPA terminates the surplus and ~24 GiB of requests is released immediately; reporting/invoices schedule within a minute or two. This is first because it is reversible in one kubectl patch and unblocks revenue reporting without a code change.
2. Right-size from data - run VPA in updateMode: Off for a week, then set requests from the p95 recommendation (typically 1.2–1.5 GiB here) with a 2 GiB limit. That restores real burst headroom instead of reserving for a heap ceiling nobody uses.
3. Then decide on nodes, once the request math is honest.
4. Safety net - give reporting/invoices a PriorityClass above the HPA-managed Deployment so a full cluster preempts batch pods rather than starving them. Verify with kubectl get events --field-selector reason=FailedScheduling.

Auto mode applies new requests by evicting and recreating pods. On a 24-replica Deployment that means a rolling eviction every time the recommender's target moves - exactly the churn an HPA exists to avoid, and visible to users as periodic capacity dips. Worse, the control loops fight: the HPA computes utilization against a request that the VPA keeps rewriting, so identical load produces different replica counts minute to minute. Running VPA and HPA on the same resource is documented as unsupported.

What I would accept: VPA in Off (recommendation only) or Initial mode. Initial sets requests once at pod creation and never mutates a running pod, so it cannot fight a live HPA - it is the right way to land a one-off right-sizing correction on existing ReplicaSets. Keep exactly one scaling actuator per Deployment: here, the HPA on CPU or a request-rate metric.

The inverse is also valid: if the workload genuinely needs a fixed replica count and only the size of each pod should change, delete the HPA and use VPA Auto - but note in-place pod resize is not the default path yet, so Auto still implies restarts. Decide which dimension is elastic, then use one tool for it.