Progressive Delivery and a Shared Admin Key
A team deploys 12 containerized services to IBM Cloud Kubernetes Service. They average 18 production deployments per week. The pipeline runs a 45-minute integration suite; 22% of runs fail on nondeterministic tests and are retried manually. Production approval is a manual gate in IBM Cloud Continuous Delivery, often approved by whoever is available. Rollback means finding the previous image tag and redeploying it by hand, which took 25 minutes during the last incident. The pipeline uses one long-lived API key with cluster administrator rights stored as a plain pipeline secret. Some runtime settings are changed in the IBM Cloud console and later overwritten by the next deploy. The team wants faster releases but is worried about breaking reliability.
1. Design a progressive delivery and rollback flow that fits the current toolchain. Include health gates and what should trigger an automatic rollback.
Read the worked answer
Build the image once, record its digest, and promote that digest through environments rather than rebuilding per stage. Deploy to a staging namespace, then canary 5% of production traffic, then 25%, then 100%, with an analysis window at each step. Health gates should use the service SLO: error rate, p95 latency, saturation, and pod restart count compared with the pre-deploy baseline. For example, roll back automatically if the canary error rate exceeds the baseline by 2 percentage points, or if p95 latency exceeds the SLO threshold for 5 minutes. Rollback should redeploy the last known-good digest and restore the previous configuration in one automated step, not a manual search through image tags. Keep database migrations backward compatible so the old version can run during rollback. The 45-minute suite should be split so fast unit and contract tests gate the canary and slower tests run in parallel, while flaky tests are fixed or quarantined rather than retried indefinitely.
2. Identify the highest-risk security and configuration weaknesses in the pipeline and how to fix them with IBM Cloud IAM and Secrets Manager.
Read the worked answer
The shared long-lived cluster-admin API key is the highest risk: any pipeline compromise or accidental command can affect every namespace and service. Replace it with separate IAM service IDs per environment and per pipeline, grant only the Kubernetes RBAC and IBM Cloud IAM roles each pipeline needs, and use short-lived credentials or rotated API keys where possible. Store secrets in IBM Cloud Secrets Manager and reference them at runtime rather than placing them in plain pipeline properties. Enable Activity Tracker to record who used which identity and when. The console drift is a reliability and security problem because the next deploy silently overwrites an undocumented change. Move runtime configuration into Git, reconcile it through the pipeline or a GitOps controller, and restrict console write access. Add branch protection and required reviews so a pipeline change cannot bypass the approval path.
3. Which parts of the release process should remain manual, and why?
Read the worked answer
Manual control should remain where judgement or accountability cannot be safely encoded, not on every routine deploy. Keep a manual approval for high-risk changes such as destructive database migrations, changes to authentication or payment flows, and releases that require customer or regulatory sign-off. Keep an explicit exception process for deploying when the error budget is exhausted. Keep a human decision for emergency rollback if automated gates are ambiguous, but give the incident commander authority to act without waiting for a change board. Everything else should be automated: routine approvals after automated gates pass, artifact promotion, health analysis, and rollback. The goal is to make the safe path the fast path, while ensuring that the few manual gates have named approvers, recorded decisions, and a bounded response time.