IA MLO: Audit request for an unreproducible production model

Audit request for an unreproducible production model

A subscription company's churn model has been in production for 14 months. An internal audit selects it and demands the exact training dataset, code, hyperparameters, and evidence that re-running the pipeline reproduces artifact churn-xgb-2024-11-03. The ML engineer finds the following: the pipeline runs from a Kubernetes CronJob using image tag :latest; requirements.txt lists pandas>=2.0 and xgboost>=2.0 with no lockfile; the random seed is drawn from time.time(); features are read from a feature table that is overwritten nightly with no history retained; the pipeline pulls a rolling 18-month window evaluated at run time. The model registry does store the artifact, a git SHA for the training repo, and metrics - but that SHA was force-pushed over a month ago and no longer resolves. The deployed model serves fine, with live AUC 0.79 against the 0.83 recorded on the registry card.

1. For each defect found, state the specific control that would have prevented it, and rank them by how much reproducibility each one buys.

Read the worked answer

Ranked by value, because they are not equal:

Data immutability (largest gap, and the only unfixable one retrospectively). A nightly overwrite means the training set for a 14-month-old run is gone. No amount of code pinning recovers it. The control is time travel on the feature table - Delta, Iceberg, or Hudi snapshots - or an explicit materialized training snapshot written to immutable storage with a manifest of partition paths plus a content hash. Read training data by snapshot ID, never by a rolling window evaluated at run time.
Image digest pinning. :latest means the code that ran is unknown. The control is to deploy by image digest, record that digest in the model registry as a first-class artifact property, and forbid :latest in any pipeline that produces a registered model. This is cheap and high value.
Dependency lockfile with hashes. xgboost>=2.0 allows a minor version bump that can change tree-building defaults and therefore the model. Control: pip-compile --generate-hashes into a fully pinned lockfile, installed with --require-hashes.
Seed recorded as a run parameter. A time-derived seed is unreproducible by construction. Control: seed from config, logged as a run parameter, and multiple seeds per run if you want to report variance.
Immutable git history. Force-pushing a branch that produced a registered model severs provenance. Control: protect the release branch, tag every model-producing commit, and record the tag alongside the SHA in the registry.

Note the ordering: pinning the image and dependencies lets you reproduce the code, but with no data snapshot you still cannot reproduce the model.

2. The audit needs an answer this week. The original dataset is gone. What do you actually deliver, and how do you characterize it honestly?

Read the worked answer

Deliver a provenance reconstruction with an explicit confidence statement, not a claim of reproducibility.

What can be recovered:

Partial data reconstruction. If upstream source systems retain raw events (orders, sessions, support tickets) even though the feature table does not, the 18-month window can be rebuilt approximately. It will not be identical - late-arriving events, retroactive corrections, and backfills will differ - so state the reconstruction method and quantify the difference by comparing feature distributions and row counts against whatever metrics were logged at training time.
Code reconstruction. The force-pushed SHA cannot be resolved from the remote, but a developer's local clone, a CI cache, or a container image built from that commit may still exist. Search CI artifacts before concluding the code is lost. If found, pin the image by digest immediately.
Statistical reproduction. Retrain N times (N >= 10) using the reconstructed dataset, best-effort code, and varied seeds. Report the mean and standard deviation of AUC. If the observed spread straddles 0.83, the claim 'a re-run yields a model of this quality' is defensible; the claim 'a re-run yields this model' is not, and should not be made.

Characterize honestly: separate 'reproducible bit-for-bit' from 'reproducible statistically' in the audit response, state which one is being provided, and list the missing evidence. An audit finding of 'no reproducibility controls existed' is survivable; a false claim of reproducibility discovered by the auditor is not.

Separately, explain the 0.79 versus 0.83. A 4-point gap over 14 months is consistent with both normal drift and a training/serving bug. Run the online/offline score comparison before concluding it is drift.

3. What is your rollback plan if the next retrain produces a model that behaves badly in production, given that the pipeline has never had to roll back before?

Read the worked answer

Rollback should be a registry pointer change, and today it is not - that is the thing to fix first.

Immediate design: the registry must hold, for every deployed model, not just the artifact but the serving image digest, the feature snapshot ID, the config hash, and the threshold/calibration parameters. Rollback then means re-pointing the serving layer to the previous deployment record and restarting pods against a pinned image. Target: under 15 minutes, exercised at least once per quarter as a drill, because an untested rollback path is a plan, not a capability.

Before any rollback is needed: keep the incumbent deployment object alive and warm through the canary period. Scaling the old model back up from zero during an incident adds cold-start minutes to your recovery time at the worst moment.

Deciding to roll back: commit to thresholds in advance and automate them - a drop in live AUC below a fixed floor for 24 hours, prediction distribution PSI above 0.2 against the incumbent, or a business guardrail (churn-saving offer acceptance) down more than 2% versus control. A human should not be deciding at 3 a.m. whether a 0.5-point dip is noise.

The hard part with a 14-month-old model: if the previous model's serving image was also pulled from :latest, rollback is not guaranteed to restore the old behaviour either. Verify the incumbent's image digest is pinned and cached before you promote anything new. If it is not, the safe move is to halt the retrain rather than to ship without a verified exit.