Operator Upgrade Failure and Multi-Tenancy on a Cloud Pak for Integration Platform

Diagnose the reconciliation failure before touching anything else. Stopped reconciliation after an operator upgrade almost always means the operator is no longer compatible with the custom resource it manages, or the operator itself cannot start. Check, in this order: operator pod status and logs in the operator namespace; the subscription's installed channel and the CSV's phase; the MQ custom resource status conditions and events for a version or field rejection; and whether the operator's upgrade changed the required channel for your installed queue manager version. Also verify that the storage classes the queue managers reference still resolve and that any webhook is healthy. Do not delete and recreate queue managers to force reconciliation - with persistent messages, that is how you lose data or stall production queues.

Sequencing the upgrades:

1. Establish the supported combination. Look up the compatibility matrix for your exact Cloud Pak for Integration release and the target OpenShift minor version. Never upgrade either side on the assumption that it will probably work.
2. Do not skip channels. Operators must be upgraded through the intermediate channels specified for your release; jumping from an old channel straight to the latest is a common cause of exactly the symptom you are seeing.
3. Typical order is OpenShift first, then the Cloud Pak operators - OpenShift within a version supported by the currently installed Cloud Pak release, then the Cloud Pak operators and capabilities, then OpenShift again if the release notes require it to reach the final target. Confirm this against the documentation for your versions rather than adopting it as a rule.
4. Fix reconciliation before upgrading anything further. Upgrading on top of a broken operator leaves you unable to tell which change caused what.
5. Stage it. Upgrade a non-production instance of each capability first, including one MQ queue manager with realistic persistent message load, and verify the full path end to end.

Why the second instance failed. API Connect requires its own namespace for an instance; a single namespace cannot host two API Connect instances. This is a scoping constraint of the capability, not a quota problem, so adding resources will not fix it. Namespace-scoped operators still install cluster-wide resources - CRDs, webhooks, and in some cases cluster-scoped controllers - so the isolation you get is administrative and quota-based, not a second independent control plane.

Recommended model:

• One namespace per team per capability instance. Each team gets a namespace with its own resource quota, limit range, network policy, and role bindings. This gives real quota isolation and clean chargeback, and it is what the operator scoping model actually supports.
• Separate namespaces for each API Connect instance, one per environment or tenancy that genuinely needs its own API manager, analytics and gateway. Plan the persistent storage for analytics up front: the analytics subsystem is disk-hungry and does not shrink itself, so size the volume and monitor it. Running out of analytics space is a predictable, preventable outage.
• Keep shared, cluster-scoped components on the platform team. CRDs, webhooks, Platform Navigator and the operators themselves remain shared infrastructure. Teams must not own operator installations, or you will get version skew across namespaces that is very hard to diagnose.
• Do not give every team a full capability stack by reflex. Three teams wanting their own runtimes may be satisfied by separate MQ queue managers and App Connect integration servers in their own namespaces, sharing one API Connect instance and one Event Streams cluster. That is far cheaper in capacity and operational surface.
• Set quotas from measured usage, not from aspirations, and make them explicit so a team hitting a quota gets a clear error rather than a pod that never schedules.

Treat 82 percent allocation as a warning, then measure what it actually means. Allocation is a promise, not usage. Gather, per worker node, the ratio of requested CPU and memory to allocatable, and separately the actual utilisation. The dangerous configuration is low real utilisation with high requests: nothing is available to schedule, yet the cluster is idle. If requests are inflated, right-sizing them buys more than adding hardware. Also check memory separately - memory is not compressible, and a node that cannot satisfy a memory request will not schedule the pod regardless of CPU headroom.

Then check for constraints that look like capacity but are not:

• Pods stuck Pending with insufficient-CPU events confirm genuine exhaustion; pods stuck for volume or anti-affinity reasons need a different fix.
• MQ and Event Streams need storage with adequate IOPS and stable latency, not just capacity. A new tenant that cannot get a suitable storage class is a blocker regardless of CPU.
• API Connect analytics consumed the available persistent volume once already. Forecast storage growth per tenant before admitting anyone.

Before adding tenants:

1. Right-size requests and limits on existing workloads, including the four integration servers and the queue managers.
2. Add worker capacity or enable machine autoscaling, but note that autoscaling does not help stateful workloads with persistent volumes and stable-identity requirements in the way it helps stateless ones - MQ queue managers and Kafka brokers need capacity that is present, not capacity that appears later.
3. Admit tenants with quotas derived from the measured figures, and reserve headroom for node failure. Six workers at 82 percent means the loss of one node cannot be absorbed.
4. Sequence it: stabilise the MQ operator and finish the OpenShift upgrade first; admit new tenants onto a platform whose upgrade path is known, not one mid-change.