GISI ADO Level: The Hotfix That Vanished Between Releases

The Hotfix That Vanished Between Releases

Northwind Analytics runs a B2B SaaS product with 16 engineers, one repository, and continuous deployment of main to its cloud tier. On-premises customers, however, receive a monthly frozen build cut from a release/YYYY.MM branch that is cherry-picked from main by the release manager.

In March a customer on release/2024.03 reported invoice totals that were off by one cent because of a rounding bug. An engineer fixed it directly on the release branch (commit c7a1e9), tagged 2024.03.4, shipped it, and closed the ticket. Nobody merged the change back to main, and the fix shipped without an automated test.

In May the same defect reappeared in 2024.05 for on-premises customers. Separately, in April a developer force-pushed release/2024.04 after a bad merge to "clean up history", rewriting three commits that two colleagues had already branched from, which cost half a day of recovery.

main is protected with required reviews and force-push blocked. Release branches have no protection rules at all. Roughly 140 commits land on main each day.

1. Explain, using Git mechanics, why the March fix was absent from 2024.05, and give the exact commands you would run to confirm the diagnosis and to sweep for other fixes stranded on release branches.

Read the worked answer

The commit is not an ancestor of main, therefore it is not an ancestor of any later release cut point. A cherry-pick creates a new commit with a new SHA and no parent link back to the original, so c7a1e9 lives only on the ancestry of release/2024.03. When 2024.05 was cut from main, the fix was simply not in the tree - nothing warned anyone because there was no test asserting the rounding behaviour.

Confirmation commands:

git branch -a --contains c7a1e9 - expect only origin/release/2024.03.
git merge-base --is-ancestor c7a1e9 main; echo $? - a non-zero exit proves main lacks the fix.
git log --all --oneline -S 'ROUND_HALF_EVEN' -- src/billing/ - locates the change and any sibling edits touching the same logic.
git cherry -v main release/2024.03 - the systematic sweep. It compares by patch-id, not SHA, so it lists every commit on the release branch whose equivalent does not exist on main. Run it for every live release line, not just March.

Remediation: git cherry-pick -x c7a1e9 onto main so the provenance line records the source commit, add a regression test covering the cent boundaries (0.005, 1.005, negative totals), cut an out-of-band 2024.05.x, and notify affected on-premises customers with the root cause. Then repeat git cherry across all supported release branches before closing the ticket - a one-off fix without the sweep will miss the next stranded patch.

2. Design a backport and merge-forward policy plus the automated checks that would have caught this before release. Name the specific Git commands or platform settings involved.

Read the worked answer

Two viable models; pick one and enforce it mechanically.

Model A - release branches receive nothing but backports. Every commit on release/* must have a patch-equivalent on main. CI on every push to a release branch runs git fetch origin main && git cherry main HEAD and fails if any line is returned. Use -x on all cherry-picks so the source SHA is recorded in the message, which makes audits trivial. A nightly job does the same sweep and opens a ticket per stranded commit.

Model B - no unique commits on release branches at all. Releases are tags on main; old lines are serviced by tagging a hotfix branch that is merged to main within one business day. This removes the class of error, but takes a migration period, so run Model A's check during the transition.

Guardrails, regardless of model:

Apply the same branch protection to release/* as to main: required review, no force-push, no deletion, status checks required.
Require the backport check as a required status check on the release branch, not on main (where it would be a no-op).
Cut releases from a tag, never from a moving branch tip, so what shipped is reproducible.
Add a CODEOWNERS entry for release/* so the release manager is a required reviewer.

The policy only holds if the check is blocking. A documented convention that nobody verifies is exactly what produced the May regression.

3. The force-push rewrote shared history. What recovery and prevention steps do you take, and would you allow force-push anywhere in this organisation? Justify the boundary.

Read the worked answer

Recovery. A force-push moves the ref; it does not immediately delete objects. The old tip is usually recoverable from the pusher's local git reflog, from the server-side reflog if the platform keeps one (GitLab keeps refs/keep-around, GitHub exposes the old SHA through the activity/events API), or from a mirror with git fsck --lost-found. Once the old tip SHA is known, recreate a recovery branch: git branch recover/2024.04 <old-sha> and push it. Colleagues should fetch and git reset --hard origin/release/2024.04 after verifying the tree matches what they expect; anyone whose local work was based on the rewritten commits should git rebase --onto origin/release/2024.04 <old-base> <branch> rather than re-do the work.

Prevention. Enable branch protection on release/* and main: block force-push and deletion, require linear-history discipline only where it matters. Consider server-side receive.denyNonFastForwards on self-hosted Git for any branch matching release/*.

Where force-push is legitimate. Personal, single-author feature branches that nobody else has fetched. Interactive rebase to clean up a messy WIP branch before review is normal and useful. The boundary is not branch naming but shared-ness: if a second person has pushed to the branch or based commits on it, rewriting is off the table. A practical rule: force-push allowed only on branches matching feature/<initials>-* with no review comments and no other authors; everything else blocked by policy, with an audited admin override for genuine mistakes.