The Fraud Model Nobody Can Rebuild

The failure is not one missing file but three mutable references standing in for immutable facts.

• Data. The run stored an S3 prefix, not a snapshot. Because the prefix is rewritten nightly, the bytes the model saw no longer exist. What should be logged is a dataset identity - a content hash or a table version (Delta/Iceberg snapshot ID, DVC hash) - plus the query and time window used to build it.
• Code and environment. latest is a pointer, not an artifact. Logging the resolved image digest and the git commit is what pins behaviour; latest re-resolves to whatever was pushed most recently.
• Selection. best_model as a mutable tag destroys provenance. The registry model version should link to an immutable run ID, and promotion should be an event recorded in the registry, not a tag edit.

Further gaps: seeds, lockfile hash and the feature-computation code version were never logged, so even with the data the score could differ.

The change is to make every run reference content-addressed inputs, and to make promotion a registry action that captures run ID, image digest and dataset version as one immutable record. The practical test: given only the registry entry, could a new engineer rebuild the model without asking anyone? Today the answer is no at every step.

Start by sizing what is actually expensive. Params, metrics and tags live in PostgreSQL; 121,000 runs is small, single-digit GB. The 3.1 TB is artifacts - checkpoints, plots, evaluation outputs. So retention should be artifact-tiered, not run-tiered, and no run record should ever be deleted.

Proposed tiers:

• Production-stage models and their still-serving predecessors: full artifact set retained until two years after retirement, matching the compliance clock. Tag artifacts at promotion time so lifecycle rules can find them.
• Candidate and finished runs: metrics and params forever; artifacts 60–90 days.
• Failed or killed runs: artifacts 7 days, metadata forever.
• Within a run: keep the best and last checkpoint, not every epoch.

Two enablers matter more than the schedule. First, deduplicate by content hash so identical datasets and base checkpoints are stored once - sweeps that differ only in hyperparameters then cost almost nothing. Second, produce less: log summary metrics per epoch and a sampled subset of step metrics. That step-level logging is what actually slows the UI, and it is cheaper to fix than storage.

Finally, review the policy against the compliance rule explicitly. An automated lifecycle rule must never be able to touch anything reachable from a production registry version; that link should be a hold tag applied automatically at promotion, not a manual checklist item.

At 400 runs a week, the unit of comparison should be a sweep, not a run. Use one parent run per sweep that records the search space, dataset version and objective, and child runs for each trial. A comparison view then loads tens of parents rather than thousands of children.

Conventions that pay off:

• A fixed tag schema applied by the launcher, not by hand: team, owner, purpose (baseline / sweep / production-candidate), dataset_version, and the metric that decides promotion.
• Structured run names, e.g. fraud-gbt/sweep-2024-06-11/trial-017, so search and sort work without the UI.
• Log the objective as a summary value on the run and keep per-step series only for runs someone intends to inspect. Step series on every trial is what makes 40-second comparisons.
• Query programmatically through the tracking API for ranking and filtering. The UI is for the last ten runs, not the first pass.

Then register only the winner, with the registry version linking to the child run that produced it. That keeps the registry small, keeps provenance intact, and turns 'which of these 400 is best at a 0.5% alert budget' into a single query instead of an afternoon of clicking. The parent/child split also gives the sweep a natural place to record why it ran, which is exactly the context missing when someone revisits the results three months later.