DE MLO common mistakes
DE MLO · Understand the traps and how to avoid them.
Treating a local notebook that produced a good metric as the finished artifact, and only afterwards trying to reconstruct which data version, hyperparameters, and code commit were used.
Notebooks encourage iterative overwriting of cells, so the run that produced the reported metric is no longer reproducible; candidates assume the final cell state equals the recorded result.
What to do instead
Log each run to an experiment tracker at start time: parameters, dataset version or hash, code commit, environment, and metrics as they are produced. Treat the logged run ID, not the notebook, as the source of truth.
Registering a model into the model registry without linking it to the run, dataset snapshot, and evaluation report that justified it, then promoting it by copying files manually.
The registry is treated as a file store rather than a lineage record, so promotion decisions cannot be audited or rolled back later.
What to do instead
Promote by version with explicit stage or alias transitions, each pointing back to a run ID and evaluation artifact. Require a documented evaluation summary before any transition to production.
Tuning on the test set, or repeatedly checking test metrics during hyperparameter search and keeping the configuration that scores best.
Candidates want fast feedback and only hold out one split, so the test set silently becomes a validation set and reported performance is optimistic.
What to do instead
Carve out a validation split (or use cross-validation) for all search and model selection. Touch the test set once, at the end, for the final unbiased estimate.
Using a random train/test split on time-ordered or grouped data, such as transactions or per-patient records.
Default splitting utilities shuffle rows, leaking future information into training or putting the same entity on both sides, which inflates offline metrics.
What to do instead
Split by time with a cutoff for temporal data and by group key for repeated entities. Verify no overlap of entities or future timestamps across splits.
Reporting a single aggregate metric without segment breakdowns, confidence intervals, or comparison against a baseline.
Aggregate numbers are easy to produce and look convincing, so candidates skip the analysis that would reveal a model that only helps the majority segment.
What to do instead
Report per-segment metrics, a naive baseline, and uncertainty (bootstrap intervals or repeated splits). State which segment the model is and is not fit for.
Serving a model with preprocessing reimplemented in the serving code path, diverging from the training-time transformations.
Training code and serving code are written by different people or at different times, and the transformation logic is duplicated rather than shared.
What to do instead
Package preprocessing with the model (a pipeline object or a versioned transform artifact) so the same code executes in training and serving. Add a parity test comparing offline and online feature outputs for the same raw input.
Choosing a deployment pattern without considering latency, throughput, and update cadence, for example deploying a large model as a synchronous real-time endpoint when batch scoring would suffice.
Candidates default to the most familiar pattern (REST endpoint) and treat serving infrastructure as an afterthought rather than a requirements-driven decision.
What to do instead
Derive the pattern from stated requirements: batch for high-volume offline scoring, real-time for low-latency interactive use, streaming for continuous events. Justify the choice against measured latency and cost targets.
Monitoring only infrastructure health (CPU, memory, uptime) and treating a green dashboard as evidence the model is working.
Standard observability tooling is readily available and gives immediate signals, while model-quality monitoring requires labels and deliberate design.
What to do instead
Add model-specific monitors: input feature drift, prediction distribution shift, and delayed-label performance. Define alert thresholds and a review cadence tied to how quickly labels arrive.
Assuming a fixed input schema and failing silently on new categories, nulls, or out-of-range values instead of detecting and handling them.
Production data drifts from the training distribution, and serving code often coerces or drops unexpected values without logging.
What to do instead
Validate inputs against a schema derived from training data, log rejected or coerced records, and route anomalies to a review queue. Test with deliberately malformed and unseen-category inputs.
Describing reproducibility as 'we use Git' without pinning data versions, random seeds, library versions, or the hardware and environment used for training.
Version control covers code only, and candidates equate commit hashes with full reproducibility, missing data and environment drift.
What to do instead
Record a reproducibility manifest per run: code commit, data snapshot identifier, dependency lock file, container image digest, seeds, and hardware. Verify by re-running one historical run and comparing metrics within a stated tolerance.