The 40-million-row index change during a production deployment

At 04:30 the priority is to restore service or reach a controlled state, not to finish the deployment. First, have the DBA identify the exact blocking session and estimate completion. If the index build has a realistic finish time before 07:00, let it complete while communicating a revised ETA. If it is blocking all writes and has no ETA, kill the index build, assess database integrity, and decide between deploying the remaining modules if the catalog is consistent and the app can start, or restoring the pre-deployment snapshot and redeploying the previous tag. Do not attempt a partial rollback from LifeTime without confirming which modules were already published. Communicate to the business at 04:45 with three facts: current state, next decision point, and expected service restoration. Assign one incident commander, one DBA lead, and one communications lead. Preserve logs and the deployment report for the post-incident review. The reasoning: a stalled schema change can turn a delay into data corruption if killed blindly.

Treat the index and backfill as separate, reversible changes. For SQL Server, create the index with online and resumable options during a low-traffic window, or pre-create it before the release so the deployment only references it. For the new mandatory attribute, use expand/contract: add a nullable column, deploy code that writes both old and new shapes, backfill in batches of 10,000–50,000 rows with commits and throttling, then add the constraint and make it mandatory in a later release. Rehearse against a production-sized copy and measure duration, lock waits, and log growth. Split the OutSystems solution so schema changes deploy in a separate, earlier window from application modules. Add a pre-deployment check that fails if a script is expected to run longer than the window. The reasoning: a 40-million-row table turns a routine deployment step into a long-running lock holder; the safe pattern is to decouple schema evolution from application deployment.

Classify every release by database risk and require DBA sign-off for high-risk changes. Add a deployment rehearsal on a production clone with production data volumes, and record actual timings. Use the expand/contract pattern as the default for schema changes. Split large solutions into a database-change deployment and an application deployment, each with its own window. Define rollback criteria before the window opens: maximum lock wait, maximum elapsed time, and a named decision-maker. Instrument the deployment with a runbook that lists the exact SQL sessions to watch. After the incident, add a checklist item for long-running operation estimates and a post-deployment verification script. Review the deployment plan in a change advisory board for any release touching tables over one million rows. The reasoning: LifeTime orchestrates deployment but cannot make a 40-million-row index build fast; the control has to be in change design and rehearsal.