A Churn Model With Post-Event Leakage and a Costly Threshold
A subscription company built a churn model on 300,000 customers with a 2% churn rate. The random-split test AUC is 0.88, but precision in the top decile is 0.18 and recall at the chosen 0.50 threshold is 0.35. The retention team can contact 5,000 customers per month; a contact costs $40, and the average retained customer is worth $600 in margin. Two features are suspicious: 'account_closed_date' and 'last_cancellation_reason', both updated by the billing system after a customer churns. The model was trained on 2022–2023 data and validated with a random 20% holdout. The team wants to lower the threshold to 0.20 to 'catch more churners.' A vendor claims its model reaches 0.90 AUC. The head of data asks whether the model is ready and how to set the threshold.
1. Is AUC 0.88 evidence that the model is ready? Evaluate it against the precision/recall numbers and the business costs.
Read the worked answer
AUC 0.88 measures ranking across all thresholds and is inflated by the 2% prevalence and by leakage; it does not show the model is ready. Top-decile precision 0.18 means that if the top 10% (30,000 customers) were contacted, about 5,400 would be true churners-90% of all 6,000 churners-but contacting 30,000 is far above the 5,000 monthly capacity. At 5,000 contacts, the relevant metric is precision at 5,000. The break-even precision depends on uplift: cost $40 per contact, value $600 per retained customer. If the intervention retains 50% of true churners, break-even precision is 40/(0.5×600)=13.3%; if it retains 20%, break-even is 33.3%. Precision 0.18 is profitable only if uplift exceeds about 22%. More importantly, account_closed_date and last_cancellation_reason are recorded after churn, so they are target leakage and will be missing or stale at scoring time. The random split also leaks future information. The model is not ready.
2. Recommend a validation design and a threshold policy that matches the retention team's capacity.
Read the worked answer
Remove post-event features and rebuild using only fields available at the decision point. Validate with a time-based split: train on 2022, validate on early 2023, test on late 2023/2024, and use rolling-origin evaluation. Report precision@5,000, recall@5,000, calibration, and expected profit-not AUC alone. Set the operating point by capacity and economics: rank customers by predicted probability, contact the top 5,000, and launch only if expected profit = 5,000 × precision@5,000 × uplift × $600 − 5,000 × $40 is positive. Run a holdout experiment to estimate uplift; without it, use a conservative assumption. A 0.20 threshold is not a policy; it may select far more or fewer than 5,000 and its precision is unknown. Monitor calibration and drift monthly. The vendor's 0.90 AUC is not comparable without the same validation design and prevalence.