Fraud Analytics: Replacing Rules or Adding a Shadow Model?
A personal auto insurer receives 45,000 bodily injury claims a year. Its rules engine generates 2,500 alerts a month, but SIU can investigate only 300. Historically 18% of investigated alerts are confirmed fraud, with average recovery of $3,500 per confirmed case. A data science team proposes an ML model that, in holdout tests, puts 45% precision in its top 300 monthly alerts. Building it takes six months and $700,000; annual run cost is $300,000. The rules engine costs $150,000 a year and is understood by regulators. SIU managers fear alert fatigue and say investigators trust the current rules. The CFO wants a business case, not a model demo.
1. Build the incremental business case for the ML model versus the current rules-only process. State assumptions and the main sensitivity.
Read the worked answer
Current process: 300 investigated × 18% = 54 confirmed cases per month. At $3,500 recovery, that is $189,000 per month, or $2.268 million per year.
ML top 300: 300 × 45% = 135 confirmed cases per month, or $472,500 per month and $5.67 million per year. Incremental recovery is $3.402 million per year.
Incremental cost: $300,000 run plus $700,000 build amortized over three years at about $233,000, for $533,000 per year. Net benefit is about $2.87 million per year. Break-even requires 533,000 / 3,500 = 152 additional confirmed cases per year, or 12.7 per month. That means ML precision must exceed about 22.2% at 300 alerts, before accounting for overlap with rules.
The main sensitivity is overlap: if the ML top 300 includes most rules alerts, incremental recovery is much lower. Recovery amount, investigator capacity, and false-positive burden also matter.
2. Recommend a deployment sequence and governance controls that satisfy the CFO and SIU.
Read the worked answer
Run an eight-week shadow mode: ML scores alongside rules, but SIU continues current investigations. Compare overlap, precision, recovery, and cycle time by alert tier. Do not replace rules until incremental confirmed cases exceed the break-even threshold.
Then deploy a blended queue that ranks by expected recovery, not by model score alone. Keep human review for all adverse actions, retain rules for explainability, and monitor drift, disparate impact, and investigator override reasons. Cap alerts at SIU capacity to avoid alert fatigue.
Governance should include model risk review, documented feature lineage, and a kill switch. The CFO gets a staged investment with a measurable gate; SIU gets a tool that respects capacity.
3. If the ML top 300 overlaps heavily with the rules alerts, how should SIU allocate its 300 cases?
Read the worked answer
Rank cases by expected recovery, combining rule severity, ML score, and claim characteristics. Use the ML score to reorder within the rules alert pool rather than to add volume. Allocate slots in tiers: highest expected recovery first, then exploratory sampling of lower tiers to monitor recall.
If overlap is high, the incremental value is in prioritization and earlier triage, not in replacing the rules engine. The business case should then be re-based on reduced cycle time and higher recovery per case, not on a larger number of confirmed cases.