The Deployment That Passed Test and Broke in Prod
A team promotes a package from a test environment to production. In test, a process calls a web service at https://test-api.internal and reads a constant API_HOST. In production, the process fails immediately with a connection error. The package included the constant object but not its value, and the production constant was set to the test host. Additionally, a new record type field was added in test but the production database migration script was run after the package import, so the field was missing during import. The team also skipped regression testing because 'only a constant changed.' The release window is 2 hours and the business is already live on the old version.
1. Identify the deployment failures and the order of operations that should have been used.
Read the worked answer
Two independent failures:
- Environment-specific configuration was packaged. The constant
API_HOSTcarried the test value into production. Constants that vary by environment should be managed as environment-specific configuration, not shipped as fixed values. The production value must be set in production, and the promotion process should verify required constants before go-live. - Schema migration ordering. The package import expected a field the database did not yet have, because the migration ran after import. Migrations must run before the package import so the schema exists when the object is imported.
Correct order: apply and verify database migration → import package → set/verify environment constants → run smoke tests → release. Add a pre-deployment checklist that fails the release if constants are unset or the migration has not run.
Skipping regression because 'only a constant changed' was also wrong: the constant controlled the endpoint, so it was the highest-risk change in the package.
2. The business is live on the old version and the window is 2 hours. What is your recovery sequence?
Read the worked answer
Stabilize first, then correct forward:
- Confirm current state: is the new version partially live? Check whether the failed process instances are stuck and whether any record type is missing the field.
- Run the migration now if it is safe and idempotent; verify the field exists.
- Set the production constant to the correct host and verify connectivity with a smoke call.
- Re-import or repair the package if the import left objects inconsistent; prefer a clean re-import over manual patching.
- Smoke test the critical process end to end before announcing success.
- Communicate status to the business and record the incident.
If the migration is not safe to run live, roll back to the prior package version, restore the constant, and reschedule the release - do not leave a half-migrated schema in production. Either way, do not extend the window by skipping verification; a second bad release inside the window is worse than a delayed one.