Notebook Environment Drift and Reproducible Jobs for a Five-Person Team
A fraud-analytics team of five runs everything inside OCI Data Science notebook sessions on a VM.Standard.E4.Flex shape (8 OCPU, 64 GB). Over six weeks, scientists ran pip install --user to add xgboost, lightgbm, and a newer numpy; two notebooks now fail on import with a scikit-learn binary incompatibility, and one model trained in a session scores 0.79 AUC in the notebook but 0.74 from the job built from the same script. Every session was created from the default PyData conda environment; block storage is 200 GB and sessions are stopped nightly by platform automation. The team must hand models to a separate MLOps group that requires an exact, restorable environment, and they need a repeatable way to run preprocessing and training without a human clicking Run All. There is no internet egress at runtime, so packages cannot be pulled from PyPI inside jobs.
1. Explain the two symptoms - the import failure and the 0.79 vs 0.74 AUC gap - and state where the environment actually lives in this setup.
Read the worked answer
Where state lives: a notebook session is ephemeral compute with persistent block storage mounted as the user home. Package installs made with pip install --user land in ~/.local/lib, which is on the Python path ahead of the conda environment's site-packages. So the 'default PyData environment' is a label, not a guarantee: five scientists sharing a home-storage model effectively have five different environments, each with a shadow copy of numpy/scipy.
Import failure: upgrading numpy under an existing scikit-learn wheel breaks the compiled ABI the wheel was built against. It is a version-skew symptom, not a data problem, and it will recur after every ad hoc upgrade until the shadowed packages are removed.
AUC gap: the job does not run in the session's mutated environment - it runs from the environment slug recorded at job creation, which is the unmutated base. Model library versions, default hyperparameters, and numeric behaviour differ, so a reproducible drop is expected. Two other causes must be ruled out before blaming the environment: a scaler or encoder fitted on the full dataset in the notebook (leakage) versus fit on train only in the job, and a different row order or seed. The diagnostic is to diff pip freeze between session and job run log, then re-run the job with the identical frozen set; if the gap persists, the bug is in the code path, not the packages.
2. Design the environment and handoff mechanism MLOps should receive. Compare conda publishing with a container image.
Read the worked answer
Session hygiene first: create one named conda environment per project, never install with pip --user, and remove existing ~/.local shadowing so the ABI conflict disappears.
Publishing path: export the exact environment (conda pack or the environment publishing workflow in the console) and write the artifact to a versioned Object Storage prefix, e.g. envs/fraud/2024-06-11/. Reference it by environment slug on jobs and pipelines. The slug is the contract: if a job does not reference it, the job silently uses the default and the reproduction fails. For the MLOps handoff, record the slug or artifact URI, a lockfile, and the Python version in the model's provenance metadata in Model Catalog.
Conda vs container: conda environments are faster to build, lighter to store, and integrate natively with sessions and jobs; they assume Python and do not capture OS-level libraries. A container image on OCIR gives bit-level control, supports non-Python dependencies (CUDA/cuDNN versions, native OCR or image libraries), and is the better immutable artifact when the team has a Dockerfile and registry. Choose containers if the model will run under a serving stack that differs from the job environment; otherwise conda plus a lockfile is sufficient and cheaper to operate.
Either way, the environment must be built once, stored immutably, and referenced by version - never rebuilt at runtime, which this no-egress tenancy prohibits.
3. Replace the manual 'Run All' with something repeatable, and say what will be lost when the job finishes.
Read the worked answer
Refactor: parameterize the script (input prefix, output prefix, fold, seed, environment slug) so it runs unattended, then create an OCI Data Science job and job runs against the published environment. Schedule via a pipeline when preprocessing and training must be chained, so step ordering and failure handling are explicit rather than implicit in notebook cell order.
Storage reality: job compute is ephemeral. The block storage attached to a job run is scratch that disappears with the run, and the same is true of anything written outside the job's designated output prefix. Every artifact that matters - model weights, metrics JSON, confusion matrices, the frozen environment spec, validation predictions - must be written to Object Storage as the run proceeds, not at the end. A run that dies at 95% with artifacts only written at the end gives you nothing, which is exactly the failure the vision team is hitting elsewhere.
Checkpointing: write checkpoints every N epochs to a run-specific prefix so a retry resumes rather than restarts. Keep the run OCID in the artifact path so any model can be traced back to a log.
What to verify after the first unattended run: that it produces the same AUC as the accepted notebook run within an agreed tolerance, and that its environment slug matches the pinned one. If not, the migration moved the bug rather than fixing it.