Auto Claims Triage: High AUC, Tight Capacity
A regional auto insurer receives 120,000 claims a year. A vendor model scores each claim for probability of becoming complex, defined as ultimate severity above $25,000. In holdout testing, AUC is 0.82. The top decile has precision 0.31; the overall complex rate is 12%. The claims team has 40 adjuster review slots per day, 250 working days a year. The model refreshes daily, but police reports and medical records that drive the strongest signals arrive about three days after first notice. A pilot using the top decile caused a 1.5-day backlog, and adjusters complained about reviewing simple claims. The COO asks whether to deploy, and if so, at what threshold.
1. What threshold should you deploy at, and what workload and capture can you defend? Show the arithmetic and explain why AUC alone is insufficient.
Read the worked answer
Capacity is 40 × 250 = 10,000 reviews per year. The top decile is 12,000 claims per year, which is 20% above capacity, so deploying at the top decile guarantees a backlog.
At a top-8% cutoff, volume is 9,600. If precision is about 0.30, that yields 2,880 true complex claims. The overall complex count is 12% × 120,000 = 14,400, so capture is 20%. A defendable deployment is tiered: top 5% (6,000 claims) for immediate adjuster review, next 5% for automated workflow, and no review for the rest.
AUC is a ranking statistic; it does not tell you precision at a capacity-constrained cutoff. With a 12% base rate, a high AUC can coexist with many false positives. You need the precision-recall curve and the cost of a missed complex claim versus a wasted review.
2. How should you handle the three-day data latency and the pilot backlog without losing the model's value?
Read the worked answer
Use two-stage scoring. A day-1 model uses only first-notice data to route claims to fast-track, standard, or reserve review. A day-3 refresh re-scores when police and medical data arrive. Do not hold claims waiting for the refresh.
Reserve about 20% of daily review slots for day-3 re-scores. During the pilot, cap the queue at 40 reviews per day and route overflow to a waitlist rather than letting it age. Track backlog, review-to-complex conversion, and leakage, meaning complex claims that were not reviewed. This preserves early action while letting stronger signals reprioritize.
3. Define the 90-day pilot success criteria and monitoring plan.
Read the worked answer
Success criteria at fixed capacity: capture at least 18% of complex claims, keep review precision above 0.28, hold backlog under 0.5 day, and reduce leakage of high-severity claims by 10% versus baseline.
Monitor score drift, precision by channel and region, adjuster override reasons, and cycle time. Kill or recalibrate if precision falls below 0.22 or backlog exceeds one day for two consecutive weeks. The key is to judge the model on decisions and workload, not on AUC.