The Algorithm That Declined the Wrong Postcodes
A personal lines insurer deploys a machine-learning model to triage motor claims for fraud investigation. The model flags 8% of claims, and flagged claims are routed to a specialist unit. An internal review finds that the model's false-positive rate is 22% and that it relies heavily on postcode and payment-history variables. Two postcodes with high proportions of lower-income and minority residents account for 31% of all flags but only 9% of claims. Flagged customers face delays of up to 40 days and, in some cases, are asked for extensive documentation. The model was procured from a third-party vendor and the insurer has no documentation of how the variables were selected. The claims director argues the model 'works' because it catches fraud, and that the disparity simply reflects where fraud occurs.
1. Analyse the fairness and compliance risks in the model's use, and assess the claims director's justification.
Read the worked answer
The claims director's justification confuses correlation with causation and ignores both fairness and legal risk. A false-positive rate of 22% means nearly one in four flagged claims is innocent, and those customers bear a 40-day delay and onerous documentation demands. The concentration is stark: two postcodes produce 31% of flags but only 9% of claims, which is strong evidence that postcode is acting as a proxy for income and ethnicity rather than for fraud. Using protected-characteristic proxies in a way that produces disparate impact exposes the insurer to discrimination and conduct risk, and the lack of vendor documentation means the insurer cannot demonstrate that the model is explainable or that its variables were selected on legitimate grounds. The director's claim that the model 'works' is unsupported: catching some fraud does not justify a 22% error rate concentrated on vulnerable groups. The insurer also cannot rely on the vendor to own the risk - as the deployer, it is accountable for the outcome.
2. Recommend the steps the insurer should take to remediate the model and its use, including governance and customer-treatment measures.
Read the worked answer
Remediation should proceed on three tracks. First, model: suspend reliance on postcode and payment-history variables until their predictive value is demonstrated independently of protected characteristics, obtain full vendor documentation, and re-test the false-positive rate by demographic group. Second, process: introduce human review before any claim is routed to the specialist unit, cap the delay for flagged claims, and ensure customers are told what is needed and why. Third, governance: establish a model risk committee with clear ownership, require documented validation before deployment and periodic fair-lending-style testing, and record the board's accountability for third-party models. Customers already affected should be reviewed, delays compensated where appropriate, and complaints monitored for patterns. The insurer should also consider whether the 8% flag rate is proportionate at all, given the error rate, and whether a narrower, explainable rule set would achieve similar fraud detection with less harm. Remediation should be reported to the regulator if the disparity constitutes a breach of conduct rules.