The Friday Afternoon Reminder Bug

The difficulty was created by four compounding choices, not by bad luck:

1. No isolation between changes. The reminder job and the UI refactor occupied the same release, so a defect in one could only be fixed by removing the other. One deployable unit contained two unrelated changes with different risk profiles.
2. No way to localize the cause. 23 commits directly on main with no tests means the defect's origin is found by reading code, not by watching a test fail.
3. Deploy coupled to release. Shipping the build and exposing the feature were the same act, so the team had no option that was both 'fix the bug' and 'keep the UI.'
4. Worst-case timing. A Friday-afternoon deploy leaves about one hour before the 6 p.m. reminder run and no business hours left to recover before the weekend, when clinics are open.

Preventive practices: short-lived feature branches merged through a reviewed pull request so each change is reviewable and revertible on its own; a smoke test suite in CI; a staging environment with a seeded copy of reminder data; deploying Tuesday–Thursday; and feature flags so deploying the UI code is not the same as turning it on.

Keep it light - process overhead is a real cost for four engineers.

Branching

• main is always deployable and is a protected branch: no direct pushes, one approving review on each pull request.
• One short-lived feature/<ticket> branch per change, rebased or merged within a few days. No long-lived develop branch, no release trains.

Continuous integration

• Every PR triggers build + lint + unit tests + a smoke test against a staging database.
• Merge to main auto-deploys to staging. Staging gets the same reminder scheduler with a compressed clock so timing bugs surface in minutes rather than at 6 p.m.

Release

• Promotion from staging to production is a separate, manual button pushed Tuesday through Thursday mornings.
• Risky user-facing changes sit behind feature flags, defaulting off, so code can ship before the feature is announced.
• Tag every production deploy (prod-YYYY.MM.DD-n) so there is always a named commit to return to.

Hotfixes

• hotfix/<ticket> branches from the production tag, gets a review and a targeted test, deploys immediately regardless of day, and is merged back to main.

Guardrails

• Never deploy on Friday afternoon or the day before a holiday unless it is a hotfix.

Cut a hotfix; do not revert or roll back.

• git revert of the offending commit is only clean if the reminder change is a single self-contained commit. With 23 mixed commits and no tests, they cannot be confident which one is responsible, and reverting the wrong one re-breaks production with less time to fix it.
• A full deployment rollback restores the previous build, which also removes the UI refactor two clinics are actively using - trading a scheduling defect for a workflow disruption.

Procedure:

1. Branch hotfix/reminder-timing from the current production tag.
2. Determine the correct timing behaviour from the ticket and the clinics' description, not from the broken code's intent.
3. Add one automated test that asserts the expected send time; it should fail before the change and pass after.
4. Review, merge, deploy the hotfix branch, tag it.
5. Verify against a real scheduler run or a staging replay before 6 p.m., and watch the first production run.
6. Merge the hotfix back into main so the fix is not lost at the next release.

If the cause cannot be found in time: roll back and immediately re-deploy only the UI work behind a flag. A rollback plus a controlled re-release is a legitimate second choice - but a silent rollback that yanks a working feature is not.