Statistical and Machine Learning Methods
What the exam tests
This domain tests method selection, validation design and honest reporting of performance.
Core methods
- GLMs are interpretable, with link functions and distributional assumptions; regularisation such as ridge, lasso and elastic net handles many predictors.
- Trees and ensembles: random forests reduce variance; gradient boosting reduces bias; both need tuning and calibration.
- Neural networks are flexible but data-hungry; they require careful validation and explainability tools.
- Unsupervised methods include clustering and PCA for segmentation and dimension reduction.
- Survival models handle time-to-event outcomes such as lapse or death.
Rules
- Bias-variance trade-off: more flexible models fit training data better but may generalise worse.
- Use k-fold cross-validation for tuning; keep a final holdout untouched.
- For frequency use Poisson deviance; for severity use gamma deviance; for binary outcomes use log loss or AUC; for ranking use lift or Gini.
- Discrimination and calibration are different: a model can rank well but be miscalibrated.
Worked example
A claims frequency model compares a Poisson GLM with gradient boosting on 5-fold cross-validation. The GBM has lower mean Poisson deviance (0.412 versus 0.431), but its predicted aggregate claim count is 8% below the observed count. The team calibrates the GBM with an offset or scaling factor, rechecks deviance and calibration on the holdout, then compares interpretability and operational cost. If the gain is small and the regulator requires reason codes, the GLM may be the better business choice.
Traps
- Tuning on the test set.
- Using accuracy for imbalanced binary outcomes.
- Reporting AUC without calibration.
- Forgetting that cross-validation must respect time order for temporal data.