Manage and Monitor Models and Experiments
Experiment tracking and reproducibility
Managing ML means knowing what produced every result. Understand experiment tracking: logging parameters, metrics, artifacts, and code versions per run so results can be compared and reproduced. Know how to organize runs into experiments, tag baselines, and register the winning artifact as a model version. Reproducibility requires capturing data version, environment, and random seeds, not just hyperparameters.
Monitoring in production
Candidates must distinguish data drift (input distribution changes), concept drift (the input-to-output relationship changes), and prediction drift. Know that label-based metrics lag because ground truth arrives late, so proxy signals such as input statistics, prediction distributions, and feature null rates are monitored continuously. Understand scheduled retraining triggers, model refresh cadence, and the difference between retraining on a fixed schedule and retraining when drift thresholds are breached.
Traps to watch for
- Monitoring only accuracy, which is unavailable in real time for many problems.
- Confusing data drift with concept drift; drift in inputs does not always degrade performance.
- Retraining automatically on drifted data without validating label quality.
- Losing reproducibility by not recording data versions or seeds.
- Treating a one-time evaluation as ongoing monitoring.
Worked example
If the input distribution of a key feature shifts sharply but offline evaluation on recent labeled data shows stable performance, the change is data drift without concept drift; alert and investigate rather than immediately retraining.
Rule of thumb
Monitor inputs and outputs continuously, labels as they arrive, and performance on a delayed cadence; every alert should map to a documented action.