Daily Releases, a Monolith, and a Change Freeze

Answer to the VP: deploy frequency is an outcome, not a target. Committing to "daily" as a goal pushes the organisation to run the same risky batch more often, which raises the change failure rate rather than lowering it. The precondition is making a single change small and reversible, then frequency follows.

Must be true first:

1. Deploy is decoupled from release. Changes ship dark behind flags and are enabled independently. Without this, "daily deploy" means "daily customer-visible change", which is a much harder problem and not the one to start with.
2. Database changes follow expand-contract and every migration is backward compatible with the previous application version. The NOT NULL incident is the proof this does not currently hold.
3. Rollback is automated, exercised, and fast. "Restore from snapshot" is disaster recovery, not rollback - four hours and data loss means nobody will ever choose it under pressure, so in practice the team forward-fixes live. Run a rollback game day quarterly and measure the time.
4. Progressive delivery exists beyond one service. Canary or blue-green with automated analysis of error rate and latency at each step.
5. Observability at the user-journey level. If nobody can see failed checkouts within minutes, small frequent changes are riskier than big weekly ones.
6. Flag hygiene. 140 flags with 30 unevaluated is an inventory problem that also breaks rollback, because you cannot reason about which code paths are live.

Proposed sequence: get the monolith to twice-weekly with automated rollback and expand-contract migrations in one quarter; prove it on a lower-risk service first; then move to on-demand deploys for flag-gated changes. Report deployment frequency, change failure rate and time to restore together - frequency alone is not evidence of improvement.

What went wrong. The release combined a destructive constraint with an application deploy in one window. The backfill happened during the window, so the column was populated - but the column had no default, and the previous application version wrote INSERTs that did not name it. Rolling back the application therefore produced a schema the old code could not satisfy. The rollback plan and the migration plan were written by different people against different assumptions and never tested against each other.

The rule: rollback means rolling back the application, and forward-fixing the database. A database cannot be rolled back safely once it has accepted writes, so every migration must be compatible with both N and N-1 application versions for the duration of the rollout.

Expand-contract for the same change:

1. Expand: add the column as nullable, with no constraint. Deploy. Nothing reads or writes it yet. Reversible by a no-op.
2. Dual-write: deploy application code that writes both the old and new paths, and backfill in batches out of band, idempotently, with progress and throttling. Monitor replication lag and lock contention.
3. Switch reads: deploy code that reads the new column, with a flag to revert to the old read path. Verify with a data comparison job.
4. Contract: only after all instances are on the new code and stable for a defined period, add the NOT NULL constraint and drop the old column - as a separate, later change, ideally in a low-traffic period, using a validated constraint rather than a blocking table rewrite.

Verification gate: CI must run the N-1 application image against the N schema and execute its write paths. That single test would have caught this incident.

Recovery time: with expand-contract, rollback is a flag flip or a redeploy of the previous image - minutes, not four hours, and with no transaction loss. Snapshot restore remains only for true data corruption.

Connection: both are symptoms of the same defect - the inability to separate what is deployed from what is active.

The freeze. A month-long freeze exists because the company believes a change made in December cannot be made safely. That belief is rational given a four-hour, data-losing rollback. The freeze therefore protects revenue by stopping delivery entirely; the cost is a month of deferred work that lands in January at elevated volume, right after the team has lost practice. Attacking the freeze directly ("we should not freeze") is politically impossible and technically wrong until rollback improves.

What is achievable this year: a reduced-risk change policy for the freeze period - flag flips with an owner and a kill switch, configuration changes with review, and no schema migrations or dependency upgrades. That keeps small fixes flowing without betting the sales event on an untested rollback. It requires the flag system to actually be trustworthy, which is the link to the second issue.

The 140 flags. Stale flags cause three concrete release problems: the test matrix cannot cover all combinations, so behaviour in production is effectively untested; dead code paths hide defects and security issues from scanners; and, critically, you cannot use flags as a rollback mechanism if you do not know which flags are live. An unevaluated flag is either dead weight or an unknown production behaviour - both are release risk.

Proposal:

• Classify flags into release flags (short-lived, must expire), ops/kill switches (long-lived, need an owner and a runbook), and entitlement flags (permanent, treat as config).
• Add required metadata at creation: owner, type, expiry, removal ticket.
• Automate a weekly report of flags past expiry; removal is a normal ticket, not a project.
• Target: no release flag older than 90 days. Track the proportion of deploys that used a flag flip as the rollback mechanism - that number is the real measure of whether progressive delivery is working.