Tag Drift Between Dev, Stage and Prod

Root cause: latest is a mutable pointer resolved at deploy time. Each environment's pipeline resolves whatever latest means at the moment its stage runs, so the artifact identity is decided by whoever pushed most recently, not by which build passed validation. Dev and stage drifted to 815 because they ran after the 815 build; prod stayed on 812 because it ran earlier. The rollback re-deployed a pointer, not a build.

Target flow:

• Tag every image with immutable identifiers only: $COMMIT_SHA-$BUILD_ID, optionally plus a semver tag. latest may exist as a convenience alias but no deployment stage may reference it.
• Model the image as a versioned DevOps artifact: each Deliver Artifact stage produces a new artifact version tied to a specific build run, and each deployment stage takes the artifact version as a pipeline parameter (${ARTIFACT_VERSION}) rather than a hard-coded value.
• Promotion is a deployment run that carries the same artifact version from dev to stage to prod, with the approval stage in front of prod. Nothing is rebuilt on the way.
• Emit the commit SHA and build run OCID as build output variables and pass them into the deployment, so the prod record shows the exact source revision.
• Separate repositories per environment are optional - tags and versions are enough unless you need different IAM boundaries - but enforce an OCIR cleanup rule: 400 tags at 620 MB is roughly 250 GB per repository, so keep the last N builds plus everything currently deployed to prod.

Rollback is a re-deploy of a known-good immutable version, not a rebuild. Because images are never rebuilt between environments, the previous prod version already exists in the registry.

Procedure:

1. Run the prod deployment pipeline with ARTIFACT_VERSION set to the last version that passed prod - typically the one from the previous deployment run, which is recorded in pipeline history.
2. Pre-authorize the on-call group for rollback runs, or maintain a dedicated rollback pipeline with a single approval, so approval does not consume the 10-minute budget. The forward path can keep its full approval chain; the reverse path should not.
3. Retain the last ~20 deployed versions with a registry cleanup exclusion, so a rollback target cannot have been garbage-collected.
4. Keep the previous Kubernetes manifest as a versioned artifact tagged with the same $BUILD_ID, otherwise a rollback restores the image but not the desired state.

The hidden dependency: rollback only completes in 10 minutes if the schema is backward compatible. If the current schema cannot serve the previous image, rollback becomes a data restore and the RTO is fiction - which is why schema changes must be additive and decoupled from the app deployment.

Pinning matters, but it is not what caused this incident. Compliance asks for no rebuild between environments, which immutability satisfies even when the build is not bit-for-bit reproducible. Fix the priority order accordingly.

What to pin anyway:

• Base images by digest (node@sha256:...), not by floating tag.
• Remove apt-get upgrade from the Dockerfile; use a locked dependency set and an internal mirror with pinned package versions.
• Record the build inputs (base digests, lockfile hash, commit SHA) as artifact metadata for forensics.

Why bit-reproducibility still matters: after a registry loss or a security investigation you may have to rebuild and prove the output matches what ran in production. That is a different requirement from promotion, and it is worth engineering for, but it should not block the drift fix.

Rule to enforce: if a rebuild is ever unavoidable, the result is a new artifact version and it must be re-promoted through dev and stage. Never push a fresh build straight to prod on the argument that the source commit is the same - that reintroduces exactly the drift being removed, and there is no evidence the new binary behaves identically to the one that passed stage.