One project, three incompatible runtimes

Most likely cause: the deployment does not inherit the session's environment. A pip install executed in a notebook writes into that container's ephemeral layer. Nothing about it is stored in the project repository, so when the model is built, CML constructs a fresh container from the project's declared dependencies plus the selected runtime. If the notebook ran scikit-learn 1.3 pulled from PyPI while the runtime ships 1.0, preprocessing behaviour, default regularization, and even column handling can change enough to move predictions by a few percent. A re-implemented feature transform inside predict() produces the same symptom.

Changes to make

1. Move every dependency into a version-pinned, committed file (requirements.txt with == pins and ideally hashes, or a conda environment.yml). Delete the pip install cells so a missing package fails the build loudly instead of silently changing behaviour.
2. Bake the environment into a custom runtime image built from a CML base image, pushed to the internal registry, registered as a runtime, and pinned for the session, the job, and the deployment.
3. Make predict() import the same preprocessing module the notebook imports. One code path, no duplicate logic.
4. Add a golden-record test: a small fixed input file with expected outputs committed to git, executed as a job against the image and again against the endpoint before promotion. A 3% shift would have been caught on day one.
5. Record the image digest and git SHA as model build metadata so the question is answerable by inspection rather than by archaeology.

Split by runtime boundary, not by team. A project is the unit of code, runtime, and access control; putting Python 3.8/scikit-learn 0.24 and Python 3.11/PyTorch 2.x in one project guarantees eventual breakage because the runtime is a project-level choice.

• Legacy scoring project. Its own project, its own pinned image, read-only for most users, with a dated migration ticket. Keeping it running is cheaper than a rushed port, and isolating it stops it from constraining everyone else.
• New modelling project. Python 3.11 image with PyTorch baked in. Nothing is installed at session start.
• Feature pipeline. Can live with the new project, but must run inside the image that produces the feature table so the training data and the deployed transform stay aligned.

GPU pressure. Interactive sessions hold devices while idle. Move training into CML jobs with an explicit GPU resource profile so work queues and releases deterministically, time-box runs, checkpoint frequently, and schedule heavy training overnight. Request a GPU profile only for training; scoring is CPU-only. If GPUs are still scarce, ask the platform team for a dedicated YARN queue rather than letting exploratory sessions compete with production training.

The 10% failure rate. It is caused by network-dependent installs at startup. Pre-baked images remove the mirror from the critical path entirely and cut the six-to-nine-minute startup to seconds. If a build must fetch packages, point pip at an internal snapshot mirror baked into the image, and pre-pull images onto nodes. Also give each workload its own project-level RBAC so one team's changes cannot break another's environment.

Provenance has to be captured automatically at run time; anything that depends on a person remembering to type it will be missing when it matters.

Per experiment run (MLflow in CML)

• Git commit SHA of the project at run start, plus whether the working tree was dirty.
• Runtime image name and digest - the digest, not the tag, because tags are mutable.
• Hyperparameters, random seeds, and the split definition (fold boundaries or the exact WHERE clause).
• A hash of the evaluation set so two runs can be proven comparable.
• Metrics, the model artifact itself, and the ordered feature list.

Per model build

• The MLflow run ID, the image digest, and the file path of the entry point. Builds should be immutable; a rebuild is a new build ID.
• The training data identity: table names plus partition range plus, ideally, a snapshot or version identifier. Storing "we trained on risk.events" is useless six months later. Prefer immutable snapshot tables so the data has a concrete identifier.

Per prediction

• Write model_version, run_id, and scored_at into the scoring table. Without them the auditor's question is unanswerable no matter how good the experiment tracking is.

Retention

• Align retention of builds, run metadata, and scoring rows with the regulatory requirement rather than the platform default, and check that it is enforced.

Where lineage is available, register the pipeline in Atlas so table-to-table and table-to-model edges are recorded automatically rather than reconstructed from memory.