Model Building
Domain 4: Model Building
Model Building tests how you specify, train, tune, evaluate, and document models. CAP questions cover data splits, feature selection, overfitting, regularization, metrics, calibration, sensitivity, and reproducibility.
Core rules and concepts
- Split data before any preprocessing that learns parameters: training, validation, and test sets, or cross-validation. Use time-based splits when temporal order matters.
- Select features using domain knowledge plus statistical or algorithmic methods; avoid leakage from future information or target-derived fields.
- Control overfitting with regularization, pruning, early stopping, and complexity limits; monitor the gap between training and validation performance.
- Align metrics with the decision: AUC, lift, precision and recall, F1, RMSE, MAE, calibration, and cost-weighted error. A high AUC does not guarantee good calibration or business value.
- Tune hyperparameters on validation data only; keep the test set untouched until final evaluation.
- Test sensitivity to assumptions, thresholds, and data perturbations. Document model specification, features, parameters, training data, and evaluation results for reproducibility.
- Prefer interpretable models when decisions require explanation, regulation, or adoption; use post-hoc explanation for complex models where appropriate.
Worked example
A credit default model must support approve and decline decisions. The team starts with logistic regression for interpretability and a gradient-boosted baseline for accuracy. They split by application date to avoid leakage, use 5-fold time-series cross-validation, and evaluate AUC, KS, and calibration. A cost matrix shows false negatives are five times costlier than false positives, so they choose a threshold that minimizes expected cost, not the default 0.5. The boosted model has higher AUC but poorer calibration; after isotonic calibration, it still requires reason codes. The team selects logistic regression because it meets regulatory explainability and performs within the business tolerance, documenting the trade-off.
Traps that cost marks
- Evaluating on training data or tuning on the test set.
- Leakage from future variables, target-derived features, or pre-split preprocessing.
- Optimizing a convenient metric such as accuracy when the decision has asymmetric costs.
- Ignoring calibration when probabilities drive decisions.
- Overfitting with excessive complexity or undocumented manual adjustments.
- Delivering a model without reproducible specification and evaluation evidence.