Post-Certificate-Rotation 502 and a Pending Db2u Pod
At 02:10, a CP4D 4.7 production cluster on OpenShift 4.14 begins returning 502 from the Zen route after an ingress certificate rotation. Users cannot log in; Watson Studio jobs already running continue. oc get pods -n ibm-common-services shows the IAM operator Running, but platform-auth-service pods are in CrashLoopBackOff; Zen pods are 0/1 Ready. The platform uses a custom CA that was added to the ingress router but not to the CP4D truststore. A new certificate secret was created in openshift-ingress, and the old secret was deleted. One Db2u pod is Pending on a worker with a node affinity mismatch after a node was replaced. The team wants to restart all CP4D namespaces and delete PVCs to clear the issue. Explain the likely failure chain, the safe diagnostic order, and the remediation.
1. What is the likely failure chain, and why is restarting everything dangerous?
Read the worked answer
The 502 is at the route, so first distinguish router-to-service failure from application failure. Deleting the old ingress secret and creating a new one can leave the router serving a certificate whose CA is not trusted by CP4D components. IAM platform-auth-service depends on the platform truststore to validate tokens and call Zen; if the custom CA is missing there, auth-service can crash or reject tokens. Zen then has no healthy authentication backend, so its pods are not Ready and the route returns 502. The running Studio jobs continue because they do not need a new login.
Restarting all namespaces is dangerous: it destroys diagnostic state, amplifies the outage, and deleting PVCs can destroy Db2 data or catalog metadata. The Pending Db2u pod is a separate scheduling problem and should not be treated with a PVC delete.
2. Give a safe diagnostic and remediation sequence for the authentication failure.
Read the worked answer
Diagnose in dependency order. Confirm the route host and backend service endpoints; check the router certificate secret and whether the new cert chain includes the custom CA. Inspect platform-auth-service logs for x509 or token validation errors and compare the truststore secret in ibm-common-services with the ingress CA. Check Zen pod logs and readiness probes.
Do not restart everything. Update the CP4D truststore with the new CA, roll the affected deployments one at a time, and verify IAM token issuance and login. Confirm time sync, because certificate validation and tokens fail with clock skew. After login works, run a catalog preview and a Studio job smoke test. If the route still returns 502, check network policies and service ports. Record the certificate expiry and add a rotation runbook that updates router and truststore together.
3. How should the Pending Db2u pod be handled without risking data loss?
Read the worked answer
Inspect oc describe pod for node affinity, taints, and PVC node affinity. Db2u often pins pods to nodes with specific labels or local storage; replacing a node can leave an affinity rule unsatisfiable. Check whether the PVC is bound to a node or zone and whether the new worker has the required label.
If the storage is local, do not move the pod to an arbitrary node; recover or relabel the correct node. If the PVC is network storage, update the affinity or add the missing label and reschedule. Do not delete the PVC. Validate Db2 health and replication before returning it to service. Then add a pre-node-replacement checklist: drain, preserve labels, verify local storage, and test Db2 scheduling.