AZ MLO: Experiment Tracking When the Registry and the Notebook Disagree

Experiment Tracking When the Registry and the Notebook Disagree

A team of 8 data scientists trains models in notebooks on shared VMs. Each run writes metrics to a spreadsheet and saves the model file to a shared drive with a name like model_final_v3.pkl. Last month a model promoted to production turned out to be trained on a dataset that included 3 months of future data due to a join bug; nobody could reproduce which notebook produced it. Two scientists are now training variants of the same model and overwriting each other's files. Leadership wants an audit trail for a compliance review in 6 weeks. There is one platform engineer and no dedicated MLOps staff.

1. What is the minimum viable tracking and registry setup that closes the audit gap in 6 weeks, given one engineer?

Read the worked answer

The audit gap is: which code, which data, which parameters, which artifact, promoted by whom, when. A minimum viable setup captures exactly those five things and nothing more.

Minimum viable stack (buy, don't build):

Experiment tracker (managed service or self-hosted open source). Each run logs: git commit SHA, dataset version/identifier, hyperparameters, metrics, and the model artifact with a content hash. Instrument a thin wrapper function so notebooks call one log_run() instead of writing to a spreadsheet. This is the single highest-leverage change and is a few days of work.
Model registry with stages (Staging → Production → Archived) and immutable versions. Promotion is a registry action that records who and when, with the linked run ID. Overwriting becomes impossible because artifacts are content-addressed and versions are append-only.
Data versioning: at minimum, record the exact query/snapshot identifier and a hash of the training set. If the join bug is the concern, the hash is what lets you prove which data a model saw.
Promotion gate: a script/CI check that refuses promotion unless the run has a git SHA, a data hash, passing smoke tests, and a sign-off field. This is the audit control the reviewers will ask about.

Deliberately out of scope for 6 weeks: full feature store, automated retraining, lineage graphs across every table. They are valuable but not needed to answer "how was this model produced."

Sequencing: week 1-2 tracker + wrapper; week 3 registry and migration of the current production model with reconstructed metadata; week 4 promotion gate; week 5 backfill audit records for models still in use; week 6 dry-run the compliance review with a real model's full trail.

2. How do you handle the existing production model whose provenance is unknown, and the two scientists overwriting each other?

Read the worked answer

Unknown-provenance model: do not pretend it has a trail. Treat it as a known exception with a documented risk decision.

Reconstruct what you can: locate the artifact, hash it, find the notebook version if it exists, and record the dataset snapshot if the join bug window can be bounded. Label the registry entry provenance: reconstructed, confidence: low with the specific unknowns.
Retrain a replacement from current, verified data under the new tracked pipeline, and validate it against the incumbent on a recent holdout. If it meets or beats the incumbent, plan a cutover before the review; if not, keep the incumbent with an explicit exception record and a remediation date.
Give compliance the exception register, not a fabricated history. Reviewers accept documented gaps with owners and dates; they do not accept silence.

Overwriting: the registry's immutable versions solve the artifact collision mechanically - each run gets its own version, so model_final_v3.pkl disappears as a concept. For the humans:

Assign each scientist a namespace or branch convention so runs are attributable.
Require the wrapper to tag runs with the author identity automatically.
Make the shared drive read-only for model artifacts; artifacts live in the registry.
Hold a 30-minute session showing the new workflow on their actual notebooks, because adoption is the real risk with one engineer and no MLOps staff. If they route around the tool, the audit trail is fiction.

3. Six weeks later, the reviewer asks you to prove a specific production prediction can be traced to its training data. What must exist, and what gap will likely remain?

Read the worked answer

What must exist for a defensible trace:

Prediction log with request ID, timestamp, model version ID, and input feature values (or a hash plus retrievable payload).
Model version → run ID link in the registry, immutable.
Run → git commit SHA and container image digest, so the exact code is retrievable.
Run → data snapshot identifier and content hash, with the snapshot still retained or reproducible from an immutable source.
Promotion record: who approved, when, what tests passed.
Retention policy long enough to cover the regulatory window - if snapshots are deleted after 30 days, the trace breaks retroactively.

Likely remaining gap: the data hash proves which snapshot, but if the snapshot was materialized by a mutable query over a live table, you can prove identity but not reconstruct contents once the source changes. The join bug that started this is exactly this class of problem. Mitigation: retain immutable copies (or partitioned snapshots) of training data for the retention window, or store the materialized training set itself. If storage cost is the objection, retain a compact sample plus the full hash and the query definition - weaker, but disclose it.

Second likely gap: predictions logged before the tracker existed have no model-version link. Scope the claim honestly: full traceability from the cutover date forward, exception register for earlier. State both in the review rather than overclaiming.