A Failed Sunday Night Release With No Usable Rollback Point

First, classify the failure before acting. Three candidates: a data problem (backfill did not complete, so code reads nulls), a code problem (the new API module is defective), or a configuration problem. The symptom - 30% of calls failing while 70% succeed - strongly suggests a subset of code paths or records, which points at data or at a partially deployed set rather than a total outage.

Options in order of reversibility:

1. Resume or repair the data migration. Check the timer's status and log first. If the backfill simply failed or is still queued, restarting it may restore service with no code change at all. This is the cheapest possible fix and should be checked before anything else.
2. Redeploy the previous versions of only the failing modules. OutSystems lets you deploy a narrower set than the full application. If the order API module is the culprit, republishing its prior version restores behaviour while leaving the rest of REL-5 in place. Risk is low if the module has no dependents that changed.
3. Toggle a site property or disable the new entry point if the failing path is behind one, buying time without redeploying.
4. Full revert to REL-4 - almost certainly wrong, see the next question.

Choose by blast radius: option 1 has none, option 2 touches one module, option 4 reverts four months of work. In a 20-minute window, prefer the narrowest action that demonstrably restores the failing 30%. Communicate the chosen action with a stated time to verify, not just the action.

Because a redeploy is a forward deployment of older content, not a transaction undo, and because REL-4 does not describe the state the system was in at 22:59 on Sunday.

Three separate problems:

REL-4 is not the current baseline. Roughly 60 modules were published directly to Production for hotfixes after REL-4 was cut. Deploying REL-4 would remove every one of those fixes, silently reintroducing four months of defects the business has already stopped reporting. You would trade a partial API failure for an unknown set of regressions, at 23:50, with no test evidence for the combination.

Schema changes are not undone. The two modules added nullable columns and rebuilt an index. A redeploy of older module versions does not drop columns or restore the old index. That is usually fortunate - additive columns are harmless - but it means the resulting system is a hybrid of old code and new schema, which is exactly the combination nobody tested.

The data has moved. The backfill timer has partially written to the new columns, and orders placed since 23:00 may exist in states only the new code understands. Rolling code backwards while data moves forward produces the classic failure where historical rows look valid to old code but were never validated by it.

What is defensible: redeploy the specific module versions that were serving traffic before 23:00, which requires knowing what those versions were. Establish that from LifeTime's deployment history for those modules, not from a tag. If that history is unavailable, the honest answer is that the organisation has no rollback capability and must stabilise forward.

Say this plainly to the escalation lead and pair it with the narrow options already listed; the objection is not to rolling back, it is to rolling back to a fiction.

Data first. Backfilling is a forward operation and is usually safe to resume; the failure was that the release depended on the backfill completing before code could serve traffic. Restructure it in three stages: deploy code that tolerates nulls and treats "not yet backfilled" as a valid state; run the backfill idempotently in batches, monitored, with a progress log and a defined completion signal; only then enable features that require the new values. Backfill in batches so a failure is resumable rather than restartable, and record per-batch checkpoints so re-runs skip completed ranges.

Release integrity.

1. Tag every deployment, including hotfixes - no exceptions. If the process permits direct publishes for hotfixes, it guarantees untagged Production content and destroys rollback. Make the hotfix path a small, tagged release instead.
2. Keep tags immutable and record the deployed tag in the change record so the prior state is always reconstructible.
3. Prefer additive, backward-compatible schema changes across two releases: add and backfill, then switch reads, then remove the old path a release later. Never combine a destructive schema change with a behavioural change.
4. Gate the release on the migration's success signal, not just on LifeTime reporting the deployment succeeded - these are different events and the gap between them is where Sunday's incident lived.
5. Rehearse in Pre-Production with a production-like data volume and a timed backfill, and record how long it took. A backfill that takes 40 minutes against real data will not finish before traffic arrives.
6. Run the deployment with an engineer watching the order API error rate for 30 minutes after the migration completes, with a named decision point for the narrow-redeploy option.

The common thread: the organisation confused "deployment succeeded" with "release is healthy". Those need to be separate, observable checkpoints.