Machine Learning and Model Development
The lifecycle and its concepts
Candidates should be able to narrate the full model development lifecycle: problem framing, data collection and labeling, feature engineering, model selection, training, evaluation, validation, deployment, and monitoring. Understand the bias-variance tradeoff and how it manifests as underfitting versus overfitting. Know why data is split into training, validation, and test sets, and why the test set must remain untouched until final evaluation. Be comfortable with core metrics and when each is appropriate: accuracy versus precision, recall, F1, ROC-AUC, and precision-recall curves under class imbalance; MAE, RMSE, and R-squared for regression. Understand regularization, cross-validation, and the role of hyperparameter tuning.
Traps that appear in questions
A common trap is reporting high accuracy on an imbalanced dataset as success - a model predicting the majority class can look accurate while being useless. Another is data leakage: scaling, feature selection, or resampling performed before splitting contaminates the test set. Questions often confuse validation with test data, or assume more complex models are always better. Watch for claims that a model "learned" a causal relationship when it only captured correlation, and for ignoring distribution shift between training and production data.
Worked example / rule of thumb
Rule of thumb: choose the metric from the cost of errors, not from habit. Example: in fraud detection with 1% positives, a model with 99% accuracy may catch nothing; recall and precision-recall AUC better reflect usefulness. Always split first, then fit preprocessing on the training fold only.