Machine Learning Concepts and Model Lifecycle
The lifecycle
Model work follows a cycle: problem framing, data collection, feature engineering, training, evaluation, tuning, deployment, and monitoring with retraining. Candidates should know what happens at each stage and which artifacts move between them (datasets, feature definitions, model weights, evaluation reports).
Bias, variance, and generalization
The central tradeoff: underfitting (high bias) fails to capture structure; overfitting (high variance) memorizes training noise and generalizes poorly. Remedies include more data, regularization, simpler models, early stopping, and cross-validation. Understand that training error alone is a poor guide; holdout or cross-validated performance is the signal.
Evaluation and data splits
Know train/validation/test roles: training fits, validation tunes, test estimates final performance once. Metrics must match the problem: accuracy misleads on imbalanced data, so use precision, recall, F1, or AUC; regression uses MAE, RMSE, or R-squared. Know confusion matrix cells (true/false positives and negatives) and how threshold changes trade precision against recall.
Drift and maintenance
Data drift (input distribution shifts) and concept drift (input-output relationship shifts) degrade deployed models. Monitoring, shadow deployment, and scheduled retraining are standard responses.
Worked example
A fraud model scores 99.5% accuracy but catches almost no fraud because only 0.5% of cases are fraudulent. Accuracy is the wrong metric; recall and precision-recall AUC expose the failure. The fix is class weighting or resampling plus threshold tuning, validated on a held-out set.
Rule of thumb
Always ask what the baseline is and which metric the business outcome depends on.