Model Evaluation and Validation
What to know
Master the confusion matrix and derived metrics (accuracy, precision, recall, F1, specificity, ROC-AUC, PR-AUC), when PR curves beat ROC (imbalanced data), cross-validation schemes (k-fold, stratified, grouped, time-series split), the bias-variance decomposition of error, learning curves, and statistical comparison of models (paired tests, confidence intervals). Understand calibration, threshold selection tied to business costs, and the difference between validation and test sets.
Traps
- Reporting accuracy on imbalanced data where a majority-class predictor scores high.
- Using random k-fold on time series, leaking the future into the past.
- Tuning on the test set repeatedly - optimistic bias.
- Preferring ROC-AUC when the positive class is rare; PR-AUC is more informative.
- Ignoring group structure (same patient across folds) causing leakage.
Rule of thumb
Match the metric to the cost asymmetry: high recall when false negatives are expensive, high precision when false positives are. Always stratify on the label for imbalanced classification.
Worked example
A fraud model flags 20 of 100 positives (TP=20), misses 80 (FN=80), and raises 180 false alarms (FP=180). Precision = 20/200 = 0.10; recall = 20/100 = 0.20. Accuracy could still look excellent if negatives dominate - a textbook trap.