IA MLO common mistakes
IA MLO · Understand the traps and how to avoid them.
Treating a feature-engineering step as data-independent and recomputing it separately at training and serving time, so the same raw input produces different feature values in the two paths.
Candidates mentally separate 'the model' from 'the pipeline' and assume small preprocessing differences (fill order, default values, categorical encoding order) are harmless, because they never test parity.
What to do instead
Define one feature-transformation artifact (fitted encoders, imputers, scalers) that is serialized with the model and loaded by both batch and online paths; add a parity test that feeds identical raw rows through training-time and serving-time code and asserts equality of the resulting feature vectors.
Reporting a single offline metric and declaring a deployment ready, without any slice-level breakdown, calibration check, or comparison against the currently deployed baseline.
Aggregate accuracy is the number candidates are most comfortable computing, and exam scenarios that mention 'the model scores 0.92' invite accepting that number at face value.
What to do instead
Require an evaluation gate that reports per-slice metrics for the business-critical segments, a calibration or threshold analysis tied to the decision cost, and a head-to-head against the production baseline on the same time window; a candidate answer that omits any of these is incomplete.
Splitting data randomly for a time-ordered problem, or fitting preprocessing on the full dataset before the split, then quoting the resulting score as evidence of generalization.
The default train/test split in most libraries is random, and leakage through preprocessing is invisible in the code because the fit call looks routine.
What to do instead
Choose the split by the data's generative structure (time-based for temporal data, group-based when entities repeat), fit every learned transformation inside a pipeline that is fitted only on the training fold, and state explicitly in the answer which split strategy was used and why.
Designing a deployment answer that names a serving technology but never addresses schema contracts, input validation, or behavior on malformed or out-of-range requests.
Candidates equate 'deployment' with 'a container behind an endpoint' and skip the interface boundary where most production incidents actually originate.
What to do instead
Describe the request/response schema, the validation rules applied before inference, the fallback behavior when validation fails, and how schema changes are versioned; treat an answer that only names a serving stack as incomplete.
Proposing retraining on a fixed schedule only, with no monitoring of input drift, prediction drift, or realized outcome quality, and no defined trigger for an off-cycle retrain.
Scheduled retraining is easy to describe and sounds operationally mature, so candidates stop there and never connect monitoring signals to the retraining decision.
What to do instead
Pair the schedule with concrete monitors (feature distribution distance, prediction distribution, delayed-label performance), state thresholds that trigger investigation or retraining, and note the cost tradeoff between retrain frequency and freshness.
Ignoring the training-serving skew that arises from batch features being computed with a different window, aggregation, or time zone than the online feature store.
Candidates assume a feature defined once in SQL is the same feature everywhere, missing that streaming and batch implementations of the same logical feature often diverge in windowing and lateness handling.
What to do instead
Specify a single feature definition owned in one place, with batch and streaming implementations validated against each other on overlapping data; include a reconciliation check that compares backfilled batch values to logged online values for the same entity and timestamp.
Describing model versioning as 'we tag the model in the registry' without linking the version to the exact training data snapshot, code commit, hyperparameters, and evaluation report.
Registry tools expose a version number prominently, which creates the impression that lineage is solved, while the metadata that makes a version reproducible is optional and often omitted.
What to do instead
Define the reproducibility record explicitly: data snapshot identifier, code commit, environment or image digest, hyperparameters, and the evaluation artifact; require that a model cannot be promoted unless this record is complete and retrievable.
Answering a rollback question by describing how to redeploy the previous model artifact, while ignoring feature-store state, cached predictions, and downstream consumers that already received new-version outputs.
Rollback is framed as a model-registry operation, so candidates overlook that features, thresholds, and consumed outputs are part of the deployed system.
What to do instead
Describe rollback as restoring the whole serving contract: previous model plus compatible feature definitions, threshold configuration, and a plan for outputs already emitted; require that the new and old versions be runnable side by side so traffic can be shifted back without a rebuild.
Treating a CI pipeline that runs unit tests as sufficient, without a stage that validates data schema, model quality against a baseline, or the packaged artifact's ability to serve a request.
Candidates transfer software CI intuition directly to ML, where passing code tests says nothing about whether the trained artifact is fit to ship.
What to do instead
Lay out the pipeline stages explicitly: data validation, training, evaluation gate against baseline and slices, artifact packaging, and a smoke test that loads the artifact and serves a sample request; explain that a failed gate blocks promotion rather than merely logging a warning.
Handling a 'the model's live performance dropped' scenario by immediately retraining, without first checking whether the drop is data drift, a pipeline bug, a label delay artifact, or a change in the consumer's decision threshold.
Retraining is the most familiar remedy, and candidates jump to it because the scenario is phrased as a model problem rather than a system problem.
What to do instead
Structure the diagnosis: verify data freshness and pipeline health, compare feature and prediction distributions to the training reference, check label arrival and join correctness, then decide between rollback, threshold adjustment, or retraining; state the evidence that would justify each branch.