Operator-Managed Platform Degrades After Onboarding a Fourth Cluster

A CPU reading of 35% does not clear the platform. The market component holds the entire supply chain graph - every entity, commodity and candidate action - in its heap, and that working set scales with entity count, not with CPU. Adding 900 pods grew the graph; the pods are killed when the container hits its memory limit, not because the nodes are out of capacity.

Confirm before changing anything:

• kubectl top pod against the container limits, sampled every few minutes. Look for a sawtooth (steady climb, kill, restart), not a flat line at the limit.
• Market and topology container logs for OutOfMemoryError: Java heap space, plus GC time. A heap spending more than roughly 20% of time in GC is undersized even if it has not been killed yet.
• Entity and action counts through the API, compared with the week before the fourth cluster was onboarded. A step change confirms correlation rather than coincidence.
• The operator's events and the platform status page for restart counts.

Then size deliberately: raise the memory request and limit for the specific component in the Turbonomic custom resource so the operator reconciles it, one component at a time, and re-check node allocatable memory and existing requests on the six workers. You cannot hand out another 8 GB to three components if the workers already request 85% of their memory. Do not try to fix this with replicas; replicas do not reduce per-instance heap pressure.

Surviving a node loss is a Kubernetes scheduling problem as much as a Turbonomic one.

• Run at least two replicas of the components users touch (API, market, topology) and give them pod anti-affinity so they do not land on the same node. Add a PodDisruptionBudget with minAvailable 1 so a node drain cannot take both down at once.
• Spread the replicas across failure domains with topology spread constraints if the workers sit in different zones or racks.
• The historian is the exception: it is stateful with a PVC. Make sure the storage class is reachable from every node (network-backed, appropriate access mode) so a rescheduled pod can re-attach. Accept that an unplanned node loss costs a restart and some in-flight data, and record that in the runbook.
• Check the operator is not pinning pods to a single node through nodeSelector or taints.
• Budget the capacity: six workers x 64 GB minus system reservations must still fit N+1 with the doubled footprint. If the replicas do not fit, the real requirement is two more workers, not replicas: 2 and hope.

What you should not do is set replicas on the stateful historian expecting resilience; it needs storage-level recovery, not another writer.

Do not begin an operator upgrade on a platform whose market and topology pods are being OOMKilled every 90 minutes. A roll interrupted mid-reconciliation is how you get a half-upgraded CRD, and switching the subscription channel back is not a supported downgrade.

1. Stabilize: apply the memory sizing change and confirm 24 hours with no container restart.
2. Qualify: check the supported upgrade path to the target 8.13.x release, since not every hop is permitted, and confirm OpenShift 4.12 is a supported platform for that version. Confirm kubeturbo 8.12.3 is compatible with an 8.13 platform - mismatched operands are a common cause of discovery gaps immediately after an upgrade.
3. Back up: export the Turbonomic custom resource, policies, groups and target definitions through the API, and snapshot the historian PVC. Record the storage class, size and access mode.
4. Upgrade: move the Operator subscription to the target channel and watch the component rollout.
5. Validate in a fixed order: all targets Validated, a known application present end-to-end in the supply chain, action counts in the expected range, historian ingestion resumed, LDAP login working, and notification webhooks still firing.

Write the rollback before you start, and make it a PVC restore plus CR re-apply, because the operator will not downgrade for you.