Machine Learning
Machine learning in risk management tests how models are built, validated, governed, and monitored when used for credit scoring, fraud detection, market surveillance, and capital estimation. The exam emphasizes model risk: a model can be statistically strong in development and fail in production.
Core rules
- Separate training, validation, and test data; use the test set once for final generalization.
- Diagnose bias–variance: high bias underfits, high variance overfits; regularization, pruning, and early stopping control variance.
- Prevent leakage: target-derived features, future information, or post-event data must not enter training.
- Use metrics matched to the decision: AUC/KS for ranking, precision/recall for fraud, calibration for PD, and stability metrics for drift.
- Document data lineage, assumptions, limitations, challenger models, and ongoing monitoring.
Worked example
A credit-default model reports AUC 0.86 in training and 0.64 on a holdout. A random forest with 200 trees has near-perfect training separation, while logistic regression reaches 0.78 holdout. The reasoning: the forest is overfitting or exploiting leakage. Check whether “days past due after application” was included, reduce feature depth, retrain with cross-validation, and compare calibration and stability. If holdout AUC remains weak, the model should not be approved for production without recalibration and a challenger.
Traps
- Reporting accuracy on imbalanced default data.
- Tuning on the test set and calling the result out-of-sample.
- Confusing explainability with causality.
- Ignoring population drift and score stability.
- Approving a model without an owner, inventory entry, and monitoring trigger.