The Rollback That Never Happened
A team deploys a payment service weekly using a pipeline that builds a container image, pushes it to a registry with the tag latest, and updates a managed container service. Last Tuesday a deploy caused a 6% increase in declined transactions. The on-call engineer tried to roll back by re-running the previous pipeline job, but it rebuilt from the current branch and redeployed the same faulty code. Recovery took 90 minutes of manual work. The team has no record of which image digest was live before the incident, and the registry has overwritten latest three times since. Leadership wants a concrete change so the next bad deploy is reversible in under ten minutes without relying on a specific engineer's memory.
1. Explain why the rollback failed and propose the minimum change that makes sub-ten-minute rollback achievable.
Read the worked answer
The rollback failed because the deployment artifact was not immutable or identifiable: latest is a moving tag, and re-running the pipeline rebuilt from the branch head rather than redeploying the previously running bits. There was also no recorded mapping from a release to the exact image digest that was live, so even a correct image could not be selected reliably. The minimum viable fix is to tag every build with an immutable identifier (commit SHA or build number), record the deployed digest per environment at deploy time, and make rollback a deploy of the last known-good recorded digest rather than a pipeline re-run. That turns rollback into a single deterministic action that any on-call engineer can perform. Add a smoke check on decline rate so the team can detect and revert quickly. This does not require a full platform rebuild; it requires immutable artifacts plus a deployment record.
2. The team also wants automated rollback on elevated decline rates. What guardrails keep that safe?
Read the worked answer
Automated rollback needs a trustworthy signal and a bounded action. The decline-rate metric should be compared against a baseline with enough volume to avoid firing on noise, and the threshold should require sustained elevation over a short window rather than a single sample. The rollback action should deploy a known-good immutable digest, not rebuild, and should be idempotent so repeated triggers do not flap between versions. Require that the previous version's health is verified before declaring success, and cap the automation so it cannot roll back past a version that itself failed. Keep a human-visible audit trail of what triggered the rollback and which digest was restored. Finally, test the automation in a staging environment with a deliberately bad deploy, because an untested auto-rollback is itself a risk during a payment incident.