From Laptop Deploys to Auditable Releases
Your team maintains a 14-service application on managed Kubernetes. Releases happen weekly, but the last three caused user-visible errors: one bad config map, one incompatible database migration, and one image tag that pointed at a stale build. The CI pipeline builds and scans images, but production deployment is a manual kubectl apply from an engineer's laptop. Average deployment takes 50 minutes, and rollback is a documented but untested kubectl rollout undo. The database is a managed PostgreSQL instance with 2 TB of data; migrations are run by hand before deployment. The product team wants daily releases within one quarter. Auditors require evidence that every production change is approved and reproducible. The team has eight engineers and can spend 15% of capacity on platform work.
1. Design the production delivery path. Explain how it prevents the three failure types and how it satisfies the audit requirement without adding a heavy gate.
Read the worked answer
Diagnosis. The three failures map to missing controls: config was not versioned with the image, the migration was not backward compatible, and the image tag was mutable. Manual kubectl apply also makes approval and reproducibility impossible.
Design. Use a GitOps delivery path. CI builds an immutable image, records the digest, scans it, and opens a change request that includes rendered manifests. A reviewer approves the pull request; the approval is the auditable change record. A deployment controller reconciles the approved commit to the cluster, so drift is detected and corrected. Admission policy rejects images not pinned by digest and rejects workloads without required labels and resource limits.
Progressive rollout. Deploy to a canary slice, for example 5% of traffic, with the same database version. Compare error rate, p95 latency, and saturation against baseline for a fixed window. Promote automatically only if thresholds hold; otherwise roll back to the previous digest. The rollback is a revert commit or a controller rollback to a known digest, not an untested command.
Why this fits constraints. It adds review and reproducibility with limited platform effort, and daily releases become possible because the path is automated.
Evidence. Store the approved commit, image digest, policy results, canary metrics, and deployment event in one record per release.
2. Design the database migration strategy for a 2 TB PostgreSQL instance. Sequence the changes so a failed release does not require a destructive rollback.
Read the worked answer
Treat schema change as a separate, backward-compatible release. Use expand/contract.
Expand. Add the new column or table as nullable, with no default rewrite that locks the table. Deploy application code that writes both old and new shapes and reads the old shape. Run the migration as a controlled job with lock timeout and statement timeout, not from an engineer laptop. Test on a restored clone and measure lock duration.
Backfill. Backfill in batches with pauses, monitoring replication lag and lock waits. Do not run a single large update during peak.
Contract. After all instances run the dual-write code and backfill is verified, switch reads to the new shape. Only then remove the old column in a later release, after a rollback window.
Rollback. Because the old code can still read the old shape, rollback is safe until contract. If the migration fails, stop before contract; do not attempt to reverse a destructive change during an incident.
Evidence. Record schema version, migration job ID, batch progress, and the release that introduced dual writes. This ties the database state to the approved change.
3. Define rollback criteria and the approval evidence you would present to auditors. Identify one separation-of-duties gap to close.
Read the worked answer
Rollback criteria should be defined before deployment: canary error rate above baseline by a stated margin for five minutes, p95 latency regression beyond a threshold, or any failed synthetic checkout. The automated controller reverts to the last known-good digest. For database changes, rollback means stopping before the contract phase and reverting application code; it does not mean reversing a destructive migration.
Approval evidence: the pull request shows the diff, the linked ticket, the approver, and the policy check results. The deployment record shows the commit SHA, image digest, canary analysis, and who approved promotion if manual. Cluster audit logs show the controller applying the change, not a human laptop.
Test the rollback. Run a game day that deploys a deliberately failing canary and measures time to restore. An untested rollback is a claim, not a control.
Gap to close: if auditors require separation of duties, the same engineer should not both approve and trigger production promotion; enforce that in the pipeline.