Vendor Model with Missing Credit and Telematics Data
A personal auto insurer has 1.2 million policy-years and 60,000 claims. The current GLM frequency model has an out-of-time AUC of 0.78 and is well calibrated. A vendor offers a gradient-boosting model with AUC 0.81 on the same holdout. The vendor model uses credit score, which is missing for 30% of applicants, and telematics, available for only 20% of policies. The vendor fills missing credit scores with a territory-level median. The business wants to deploy the new model for renewal pricing. The regulator requires that rating factors be supported, not merely predictive, and that missing-data treatment be documented. The chief actuary asks you to assess whether the AUC gain is enough to justify deployment.
1. Why might the 0.03 AUC improvement overstate the value of the vendor model for renewal pricing?
Read the worked answer
AUC measures ranking discrimination, not calibration, profit, or rating-factor defensibility. The vendor's gain may come from credit score and telematics, but those variables are missing for most policies. Filling 30% of credit scores with a territory median creates an imputed variable that may proxy for territory and other protected or correlated characteristics. The 20% telematics sample is likely selected, so its lift may not generalize to the full book.
A 0.03 AUC increase can be concentrated in a small segment with little premium impact, or it can come with worse calibration in the missing-data groups. For renewal pricing, the insurer needs accurate predicted costs by rating cell, not just a better global rank ordering. The model must also support the factors under regulatory standards, which a purely predictive black box may not.
2. Design three validation tests before deployment.
Read the worked answer
First, test calibration by decile and by missingness group: compare predicted and actual claim frequency for policies with complete credit, missing credit, and telematics versus no telematics. A model can rank well yet overpredict or underpredict a large segment.
Second, run out-of-time and out-of-geography validation, not just a random holdout. Include a stress test that removes credit score entirely and another that replaces the territory-median imputation with a conservative missing indicator. The model should not collapse or produce implausible rate changes.
Third, evaluate lift in business-relevant segments and perform a proxy/fairness review. Check whether the credit and telematics variables create disparate impact or act as substitutes for prohibited factors. Also compare the profit-weighted lift, not only AUC, because a small discrimination gain in a large segment may not cover implementation and monitoring costs.
3. Recommend deployment conditions.
Read the worked answer
Do not deploy the vendor model for all renewal pricing on the basis of the AUC gain alone. A reasonable path is a controlled pilot on the segment with complete data, used as a segmentation or diagnostic tool rather than an immediate rating factor. Require the vendor to document missing-data mechanisms and provide model explainability for credit and telematics.
Conditions: retain the current GLM as the approved baseline until the new model passes calibration, out-of-time, and proxy tests; file any rating-factor change only after regulatory support is established; monitor drift and missingness monthly. If credit score cannot be supported as a rating factor, use the model only with a missing indicator and without the territory-median imputation. The chief actuary should be able to explain the rate impact by segment, not just the AUC.