The four-line production fix with three weeks of work behind it
A team ships weekly through Dev, QA and Prod. Production runs tag v4.2.0 of solution ClaimsSuite. On Tuesday a defect is confirmed: the amount validation in ClaimsCore accepts negative values, and agents have already issued credit notes with them. The four-line fix lives in the same module. Dev, however, is three weeks ahead: a new first-notice-of-loss wizard in ClaimsUI and a database migration that adds three columns and rebuilds an index. Last month a DBA created an index directly in Prod to rescue a slow report; it exists in no other environment. QA also fails validation right now because a shared Common module was published to Dev but never tagged. The business wants the fix in production within 24 hours and does not want the wizard released yet.
1. Lay out the realistic delivery options and recommend one with its trade-offs.
Read the worked answer
Option A - promote Dev to Prod. Fastest to start, unacceptable: it ships the wizard and an irreversible schema change into an incident, and the wizard has not been through QA sign-off. Rejected.
Option B - hotfix path from the production tag. Use LifeTime to deploy the v4.2.0 tag into a hotfix environment, apply the four-line change there, tag v4.2.1, then promote through QA to Prod. This is the recommended route. Trade-offs: it requires a spare environment and a few hours of environment setup; the hotfix environment's ClaimsCore is now a second line of development, so the fix must be merged forward into Dev the same day or it will be silently reverted by the next release; and anything in the hotfix environment that was not in the tag must not be published.
Option C - ship the full release early. Only defensible if the wizard is genuinely release-ready and the migration has a tested rollback. Here it is not, and it couples an incident fix to feature risk.
Option D - database-only mitigation. Blocking negative amounts with a check constraint buys time and loses no data, but the UI still tells agents the value is valid, so it is a stopgap at best.
Recommendation: B, with the merge-forward into Dev tracked as a same-day task, and the constraint from D as an immediate safety net if the environment bootstrap slips.
2. What exactly do you verify in the deployment plan before promoting, given the untracked index and the untagged Common module?
Read the worked answer
Dependencies. ClaimsSuite depends on Common. If Common in the target has no version of the tag the solution expects, validation fails or you ship a mismatch. Either tag Common in the hotfix path with the version that production already runs, or confirm the production Common version is compatible and pin it explicitly in the plan. A dependency that only exists in Dev is not a dependency - it is an undeployed change.
Module scope. Read the plan list and confirm only the modules you intend to change are being published. The hotfix path is what keeps the wizard out; if the plan shows ClaimsUI with a different tag than production, stop.
Database changes. The three-column migration must not appear in this plan. LifeTime applies model-driven schema changes on deploy, so an unexpected column addition is a signal that the wrong version of the module is being promoted. Review the changes before deploying to QA, not after.
The DBA index. It is real, it is in production only, and it exists nowhere in the model. Deploying will not remove it, but the next environment built from the model will lack it and the slow report will return. Add the index to the entity in the hotfix path (or record it as a versioned script) so the fix is durable, and do not let the emergency justify leaving it undocumented.
Validation and timing. Run validation in each target, confirm the tags, then deploy off-peak with the rollback tag identified up front.
3. What does rollback actually mean here, and what would you check before declaring the fix successful?
Read the worked answer
Rollback means redeploying the previous tag, v4.2.0, to the same environment. That restores code and model-driven schema as defined by that tag. It does not undo data: credit notes written while the defect was live stay written, and a destructive schema change already applied (a dropped column, a narrowed type, a deleted index) may not be cleanly reversible by redeploying. Because of that, this hotfix should contain no destructive model changes - additive only - so that rollback is a genuine option rather than a theoretical one.
Before declaring success, check:
- the negative-amount validation rejects a real negative payload, tested in QA against the same build tag;
- the timer and BPT workloads that touch claims have run at least once without errors in Service Center, since a fix in a validation action can change what downstream processes accept;
- the audit trail shows the affected credit notes were corrected or flagged, which is a business task that a deploy does not perform;
- the merge forward into Dev is merged and a build in Dev contains the same fix, otherwise the next release reinstates the bug;
- monitoring shows error rates and response times back to baseline for one full working day.
Process lesson: the existence of a hotfix path, and the fact that the DBA's index was invisible to every other environment, are both findings to fix in the delivery process, not just in this incident.