Intermittent Deploy Failures on a Shared QA Environment
Northwind Logistics runs OutSystems 11 with Development, QA, Pre-Production and Production. Three squads totalling 12 developers merge into a shared main branch, and an Azure DevOps pipeline calls the LifeTime Deployment API to push the newest tag to QA on every merge. Two unrelated teams also deploy to that same QA environment. Over six weeks, 38% of pipeline runs fail at the deploy stage, with either "There are modules with missing dependencies" or "A deployment is already in progress". Engineers retry, and roughly half of retries succeed, so the team treats the failures as flakiness. Nothing reproduces on a developer machine. Release management now wants every Production deployment traceable to one commit and executed automatically at 22:00, but QA instability is blocking sign-off. The platform team holds LifeTime administrator rights and may create environments, but budget covers only one additional environment for the next two quarters.
1. The team calls this "flaky". Separate the failure modes and explain the mechanism behind each, including how you would confirm the diagnosis.
Read the worked answer
The two messages describe two unrelated problems, which is exactly why retries look random.
"A deployment is already in progress" is environment-level serialisation, not a defect in your pipeline. LifeTime allows a single deployment per environment at a time and does not queue requests. Three teams writing to one QA means your run races theirs. Confirm by correlating the failed run's timestamp with the other teams' deployment history in LifeTime: you will typically find a competing deployment started one to three minutes earlier and completed shortly after your failure. Retrying works because by then the lock is free.
"Missing dependencies" is a version-graph problem. OutSystems deploys a concrete set of module versions, so if any module in the set references a module version that does not exist in the target environment, the whole deployment is rejected. Realistic causes in this setup:
- A shared module was republished after the tag was created, so the tag records a reference nobody ever promoted.
- The plan marked a module "unchanged" relative to QA, but QA had drifted because someone published to it out of band.
- A module used by only one squad was never promoted at all.
Retrying appears to help because a colleague manually published the missing module in the meantime.
Diagnosis order: open the LifeTime deployment log and read the exact module and version named; reconcile the generated plan against the tag manifest; then compare with what Service Center shows as deployed in QA. Do not accept "retry and it passes" as evidence of a transient fault.
2. Design the pipeline and environment model to fix this, given that only one extra environment is affordable. Justify what that environment should be used for.
Read the worked answer
Spend the one environment on removing the shared-write bottleneck, not on adding another downstream stage.
Preferred option: a dedicated automation environment owned exclusively by the pipeline. Developers get fast feedback on merge into main; human QA still happens in the shared QA environment, but only through scheduled, tagged promotions. Only the pipeline's IT user holds Change and Deploy there, so nobody can drift it.
Fallback option: serialise at the pipeline layer using a cross-pipeline exclusive lock plus bounded retry with backoff on the "in progress" response. This fixes failure mode one only; it does nothing for dependency drift, so it must be combined with the rules below.
Rules in either case:
- One tag cut from one commit on main is the unit of deployment. Never deploy "latest".
- Deploy with dependencies, then diff the generated plan against the tag manifest and fail the pipeline on mismatch. This converts a deploy-time rejection into a fast, legible build failure.
- Revoke manual publish rights on QA and Pre-Production from humans; grant them to the pipeline identity.
- Run smoke or BDD screen tests against the automation environment before any promotion.
- Write the tag name into the change record, so the 22:00 Production run references an artefact that already passed tests rather than re-cutting from main at night.
The one-environment budget makes the ordering matter: eliminating concurrent writes buys more stability than any amount of retry logic.
3. The release manager proposes blocking any Production deployment when the QA environment contains module versions newer than the tag. Evaluate that rule.
Read the worked answer
It would deadlock the pipeline and should be rejected, but it points at a real risk that deserves a narrower control.
Why the rule fails: QA is shared and continuously receives other teams' work. There will almost always be something newer than any given tag, so the gate would fire permanently and the team would start overriding it. A gate that is routinely bypassed trains people to ignore gates.
What actually matters is not what else exists in QA, but whether the tag is a coherent, already-validated set. Substitute these controls:
- Every module referenced by the tag must exist at the recorded version in the source environment of the promotion.
- The generated plan must contain only versions recorded in the tag manifest. Anything else is either removed deliberately or the tag is re-cut.
- Automated tests must have passed on that exact tag, in an environment at least as production-like as Pre-Production.
- A named approver signs the change against the tag name.
One legitimate exception: if the tag pulls in a new version of a shared module that has never run in Pre-Production, that version is untested at scale. The answer is not to block on QA freshness but to require that shared-module upgrades reach Pre-Production through their own promotion before they are consumed.
Net: gate on the integrity and test evidence of the tagged set, not on the freshness of an environment other teams own.