The Model That Agreed With Itself
A retail bank has built an internal model to estimate operational loss frequency. Backtesting over 24 months shows the model predicted 41 events and 38 occurred - an apparently excellent fit. A validation analyst notices that the model was calibrated on the same 24 months it is being tested against, and that the three largest actual losses were excluded from calibration as 'outliers'. The model owner argues the exclusions were legitimate because those events were 'one-offs' and including them would distort the frequency estimate. The model feeds a capital calculation.
1. Assess the validity of the backtest result and the outlier exclusions.
Read the worked answer
The backtest is not a backtest. Calibrating and testing on the same window is in-sample fitting: the model has been tuned to the data it is being scored on, so a close match (41 predicted vs 38 actual) demonstrates fit to history, not predictive power. The result should be discarded as evidence of model quality.
The outlier exclusions are more serious. In operational risk, large losses are precisely the events the capital calculation exists to absorb. Excluding the three largest as 'one-offs' removes the tail from the distribution and will systematically understate the capital requirement. 'One-off' is an assertion, not an analysis - the analyst should ask whether those events share a cause (a control gap, a process, a vendor) that is still live. If the cause persists, they are not one-offs.
Note also that the model owner has an incentive to keep the capital figure low; the exclusion decision should not sit with the owner.
2. Recommend the validation approach that should replace the current one.
Read the worked answer
- Out-of-sample testing. Hold out a period the model has never seen and test on that. Report discrimination and calibration separately.
- Explicit tail treatment. Include large losses in calibration or model the tail separately (e.g. a severity distribution fitted to the body plus an extreme-value treatment). Document every exclusion with a stated, testable rationale and obtain independent sign-off.
- Sensitivity analysis. Show how the capital estimate moves if the three excluded events are included. If the answer changes materially, the exclusion is doing the work.
- Governance separation. The model owner proposes; validation challenges; an independent committee approves exclusions. No single party should both choose the data and judge the fit.
- Ongoing monitoring. Track realised losses against prediction each period and escalate persistent under-prediction.