Model Risk, Validation and Communication
Domain 4: Model Risk, Validation and Communication
CB2 tests how model risk arises, how models are validated and how results are communicated to decision-makers. It rewards specific controls and clear acknowledgement of limitations.
Sources of model risk
Model risk arises from poor data, inappropriate assumptions, implementation errors, incorrect use and weak governance. It exists even when a model is mathematically sound. Controls include model inventories, ownership, version control, peer review, independent validation and restrictions on use.
Validation
Validation is independent of the model builder. Techniques include reviewing code and formulae, back-testing against actual outcomes, benchmarking against a simpler model, sensitivity testing and out-of-sample testing. Validation should assess conceptual soundness, data suitability, implementation accuracy and limitations.
Communication
Match the communication to the audience. State the purpose, key assumptions, results, uncertainty and limitations. Show ranges or percentiles rather than false precision. Document the model so another actuary could reproduce it.
Worked example
A capital model predicts that the 95th percentile of annual claims is £10m. Back-testing over 100 historical years finds 10 years above £10m, not the expected 5. The exceedance rate is 10%, double the target. This suggests the model understates the tail; possible causes include a distribution with too light a tail, ignored parameter uncertainty or a structural change in claims. The response is to investigate, recalibrate if justified and document the limitation, not to discard the back-test because the base case looked acceptable.
Traps
- Treating validation as a one-off sign-off rather than ongoing monitoring.
- Presenting a single central estimate without uncertainty or limitations.
- Assuming a model is valid because it is complex.
- Ignoring model drift after the calibration period.
- Communicating technical output without translating it into decision-relevant terms.