Hotfix at 23:00: Which Branch Ships?
A SaaS product ships weekly from main. A critical payment bug appears at 22:40 on a Wednesday. main already contains 40 commits for next week's release, including a database migration that has not been rehearsed in production. The last release is tagged v4.7.0; a release/4.7 branch exists and matches production. The on-call engineer can patch main and deploy, or patch release/4.7 and deploy from the tag lineage. The migration author says the migration is 'probably safe' but has only run on staging. The incident commander must choose a path within 15 minutes and also decide how the fix reaches main afterward.
1. Recommend the deployment path and justify it in terms of what could go wrong.
Read the worked answer
Patch and deploy from release/4.7, not main. The decision hinges on blast radius: deploying main ships 40 unreviewed-in-production commits, including an unrehearsed migration, during an active incident at night. If the migration fails, the team is now fighting two problems - the payment bug and a schema change - with degraded attention and no rollback rehearsal. The release branch matches production, so the fix's effect is isolated and attributable.
The cost of the release-branch path is merge work later and the risk of a divergent hotfix. That cost is bounded and can be managed; the cost of a failed migration during an incident is not. 'Probably safe' is not an acceptable risk statement for a 23:00 production change.
2. Specify how the hotfix must be propagated so the two branches do not silently diverge.
Read the worked answer
The hotfix must be cherry-picked or merged back to main before the next release is cut, with a test that reproduces the payment bug committed alongside it so the fix is verified on both lineages. Tag the hotfix release (e.g. v4.7.1) so production state is identifiable. Add a check - a CI job or release checklist item - that fails the next release if release/4.7 contains commits not present in main. Without that check, divergence is invisible until it causes a regression that reappears after the next deploy, which is the classic hotfix failure mode.
3. What should the incident commander require before the fix is deployed, given the time pressure?
Read the worked answer
A short, explicit gate that fits in minutes: (1) a one-line description of the root cause and the change, reviewed by one other engineer; (2) confirmation that the change is confined to the release branch and touches no schema; (3) a stated rollback step (redeploy v4.7.0) that has actually been used before; (4) a post-deploy check that the payment path succeeds on a real transaction. Time pressure justifies a smaller gate, not no gate. The migration can wait for its normal weekly release with a proper rehearsal - the incident is not a reason to smuggle it in.