Claims Triage Model: Re-rating Residual Risk Without Tested Controls
Northbridge Mutual, a general insurer with $2.1 billion in annual premium, has deployed a machine-learning claims triage model that routes motor claims into fast-track, standard or investigation queues. The model was developed in-house and validated on 18 months of historical data. At the September risk committee, the Head of Claims re-rated the 'incorrect triage decision' risk from High (inherent) to Medium (residual), citing a new weekly sampling control in which an analyst reviews 25 randomly selected fast-track claims. The control has operated for six weeks. No sample has yet detected a misroute, but the analyst is the same person who built the model. The risk appetite statement says claims leakage above 2% of incurred cost is outside appetite. Current estimated leakage attributable to triage is 1.4%, with a confidence interval of 0.9% to 2.6%.
1. Evaluate whether the re-rating from High to Medium is defensible. Identify the evidence you would require and the conditions under which the residual rating should revert.
Read the worked answer
Conclusion
The re-rating is not defensible on current evidence. Residual risk should remain High until the control is shown to be both well designed and operating effectively.
Reasoning
- Design vs operation. A weekly sample of 25 fast-track claims is a detection control, not a preventive one. It does not stop a misroute; it estimates the error rate after the fact. Six weeks gives roughly 150 reviews, but fast-track volumes are likely in the thousands per week, so the sample gives a narrow, low-power view.
- Independence. The reviewer built the model. That creates a self-review threat: the person has an incentive to find the model is working. The control is not independent.
- Appetite. Estimated leakage is 1.4% with a confidence interval of 0.9% to 2.6%. The upper bound exceeds the 2% appetite. A point estimate inside appetite does not mean the risk is inside appetite; the board should be told the probability that leakage is above 2%.
- Model risk. Validation on 18 months of history does not cover recent shifts in claims mix or fraud patterns. Ongoing monitoring, back-testing and drift alerts are missing.
Evidence required
- Independent control testing, with sample size calculated to detect a 2% leakage rate at agreed confidence.
- Segregation: reviewer separate from model development and from claims performance targets.
- Documented model monitoring: drift, override rates, queue distribution and periodic revalidation.
- A leakage measurement method reconciled to financial records, not just sample results.
When to revert
Revert to High if independent review finds the control ineffective, if leakage exceeds 2% for two consecutive months, or if model drift exceeds agreed thresholds.
2. The CRO asks how the risk should be presented to the board given the confidence interval. Draft the key points you would include and explain the judgement behind them.
Read the worked answer
Key messages
- Report a range, not a point. Say leakage is estimated at 1.4%, with a plausible range of 0.9% to 2.6%. The upper end breaches the 2% appetite.
- State the rating honestly. The residual rating remains High pending independent control testing. Do not present Medium as fact.
- Explain what is not yet known. The sampling control is six weeks old, reviews only fast-track claims, and is performed by the model developer.
- Set a decision for the board. Ask the board to note the temporary rating, approve an independent review, and agree an escalation trigger.
Judgement
A board paper that says '1.4%, inside appetite' is technically true but misleading because it hides uncertainty and the control weakness. The CRO's obligation is to give the board the information it needs to make a decision, not the most favourable number. Presenting a range and a probability of breach is more useful than false precision. The board can then decide whether to accept the risk temporarily, restrict fast-track volumes, or require additional controls before scaling. Recommend a 60-day independent review with an interim report at 30 days, and a standing monthly metric showing leakage, override rate and sample findings.