An HPA That Cannot See, and Should Not Be Looking at CPU

HPA computes utilisation as total usage / total requests, expressed as a percentage of the request. With no requests.cpu set, the denominator is undefined, so every pod reports % as <unknown> and the controller cannot compute a ratio. It leaves the replica count untouched rather than guessing - which is exactly what is observed.

Metrics-server being healthy is necessary but not sufficient; the missing piece is the request. Fix:

resources:
  requests: { cpu: 300m, memory: 512Mi }
  limits:   { cpu: 500m, memory: 768Mi }

Size the request from observed steady-state usage (CPU here is ~100m at 20% of a 500m limit), not from the limit. A request that is far too low makes utilisation look enormous and causes runaway scaling; far too high and it never triggers.

Two related traps worth naming: changing a pod's resources triggers a rollout, so do it deliberately; and the HPA must not fight a spec.replicas managed by another controller - with a GitOps tool reconciling replicas, the HPA's scale is reverted each sync unless the field is excluded from the manifest.

No. Fixing the request makes the HPA functional, not correct for this workload.

• CPU is the wrong signal. The worker is I/O-bound on the database. CPU stays near 20% whether the queue holds 200 messages or 45,000. A 70% CPU target would require four times the current CPU usage per pod before scaling - usage that never materialises while workers wait on I/O. The HPA would sit at minReplicas through the entire incident.
• The symptom is queue depth, not resource saturation. The right signal is the thing you actually care about: messages pending, or oldest-message age. That requires a custom or external metric via the prometheus-adapter, or a KEDA ScaledObject reading the broker directly.
• Scale-out is the right dimension. Pods are single-threaded, so vertical scaling helps less than adding replicas; but the ceiling is the database's connection pool, not the HPA's maxReplicas.

Reasonable configuration: scale on queue depth with a target such as 100 messages per replica and maxReplicas: 20; scale down on oldest-message-age returning to normal rather than on the queue emptying, and use a stabilisation window (e.g. 300 s) so a transient dip does not thrash replicas.

Autoscaling exposes every fixed assumption downstream. Check, in roughly this order:

1. Database connection pool. 20 pods x pool size (say 10) = 200 connections. If the database allows 100, the new pods fail or, worse, starve the existing ones. Cap the per-pod pool so maxReplicas x pool <= dB limit, or introduce a proxy/pooler.
2. Downstream rate limits. The queue broker, an external API, or a payment provider may have per-account quotas. More workers can violate them and cause a different outage.
3. Schedulability. 16 additional pods require real node capacity and, with requests now set, the scheduler needs that capacity reserved. Without the autoscaler, new pods go Pending and the backlog does not clear - the same failure mode as the drain scenario.
4. Workload ordering semantics. If order processing must be strictly serialized per order, adding consumers can introduce out-of-order or duplicate handling; confirm the queue's delivery guarantees and idempotency.
5. Disruption and rollout settings. A PDB sized for 4 replicas (minAvailable: 3) becomes trivially permissive at 20; a surge percentage that was fine at 4 may be far too large at 20.
6. Scale-down behaviour. Without a stabilisation window, replicas will oscillate on a sawtooth queue, which itself stresses the database. Set behavior.scaleDown.stabilizationWindowSeconds deliberately.