NLP claims triage and vulnerable customers
An insurer's claims operations receive about 45,000 motor claim narratives each month. A vendor's NLP model assigns each notification a severity score and routes the top 15% to specialist handlers. In an eight-week pilot on 9,000 claims, the model achieved precision 0.71 and recall 0.88 for 'complex' claims at the chosen threshold. False negatives were concentrated among claimants writing in languages other than English and among older policyholders. The vendor reports a single validation AUC of 0.84 but provides no subgroup metrics, no drift monitoring and no explanation of features. Operations wants to expand the model to all motor claims and reduce manual review by 20%. The insurer's conduct policy requires fair treatment of vulnerable customers and human accountability for significant decisions. Two complaints in the pilot alleged that important medical information was ignored.
1. Assess whether the pilot evidence supports expansion of the NLP model to all motor claims.
Read the worked answer
The pilot does not support full expansion. AUC 0.84 is a ranking statistic, not evidence that a 20% manual-review cut is safe. Precision 0.71 means 29% of routed claims are false positives, consuming specialist time; recall 0.88 means 12% of complex claims are missed, and the concentration among non-English writers and older policyholders is a conduct red flag. Nine thousand claims over eight weeks may contain few severe cases, so subgroup estimates will be imprecise. The vendor provides no subgroup metrics, drift monitoring, feature explanations or validation access. Operations' target should be conditional: expand only to a controlled shadow mode or limited channel while subgroup performance, false-negative harm and specialist capacity are measured. Human accountability must remain with claims handlers; the model should route and prioritise, not decide. Two complaints alleging ignored medical information need root-cause review before any expansion.
2. Propose a human-in-the-loop design and a set of metrics for the expanded model.
Read the worked answer
Keep a human decision-maker for every claim; the model produces a priority score and reason codes. Route all high scores to specialist review and random-sample low scores to estimate missed complex claims. Require review of any claim where a vulnerable-customer indicator, language need or medical information is present, regardless of score. Capture overrides and reasons. Metrics: precision and recall overall and by language, age band and vulnerability; false-negative rate for severe claims; time to first decision; specialist workload; complaint and ombudsman rates; override rate and agreement. Set thresholds using harm-weighted costs, not accuracy. Monitor input drift, score distribution and subgroup performance monthly, with a pre-agreed pause trigger. Explain to handlers what the score does and does not mean; train them to challenge it.
3. What governance and vendor-contract changes should be required before any expansion?
Read the worked answer
Contract changes: right to audit and independently validate the model; full documentation of features, training data and known limitations; subgroup performance reporting; drift and incident notification; retraining and change-control approval; data provenance and retention terms; exit/portability plan. Internally, add the tool to the model inventory, assign an owner, perform a data protection impact assessment if personal data are processed, and document human oversight. Consumer-duty obligations require evidence that the tool improves or at least does not worsen outcomes for vulnerable customers. Board reporting should include false negatives and complaints, not only cost savings. If the vendor refuses transparency, do not expand.