Model Evaluation, Validation, and Interpretation
Metrics by problem type
For classification, know accuracy, precision, recall, F1, ROC AUC, and the confusion matrix. Accuracy misleads on imbalanced data; precision and recall trade off against each other and depend on the decision threshold. For regression, know MAE, RMSE, and R-squared; RMSE penalizes large errors more than MAE.
Validation discipline
Cross-validation gives a more stable estimate than a single split. Watch for the gap between training and validation scores. Evaluate the final model once on the held-out test set; repeated peeking turns the test set into a validation set and inflates optimism.
Interpretation
Feature importance ranks influence but does not prove causation and can be unstable with correlated features. Partial dependence and per-row explanations help explain individual predictions. Residual analysis reveals systematic error patterns.
Traps
A classic trap: high accuracy on a 95/5 imbalanced target with a model that predicts the majority class. Another: choosing a threshold of 0.5 by default when the business cost of false negatives is high. A third: reading feature importance as causal.
Worked example
A fraud model reports 99% accuracy but 0% recall on fraud. The data is 1% fraud, so predicting "not fraud" always wins on accuracy. Switch to precision/recall and AUC, apply class weighting, and tune the threshold to the cost of missed fraud.