Churn Model With High Accuracy but Weak Discrimination
A subscription company has 120,000 active customers and a binary churn label; 8,400 churned in the last 12 months (7%). You build a Visual Analysis in Dataiku with a random 80/20 train/test split, stratified on churn. The best quick model reports test accuracy 0.931, AUC 0.58, recall 0.04 at the default 0.5 threshold, and precision 0.61. The business wants to email the top 10% highest-risk customers. A colleague suggests switching from accuracy to F1 and lowering the threshold to 0.2. The model uses 35 features, including tenure, monthly spend, support tickets, and a 'cancel_reason' field that is populated only after cancellation. The test set has 24,000 rows. The team has not inspected the confusion matrix or lift chart. You need to advise on metric, leakage, and next experiment.
1. Which evaluation metric should drive model selection and threshold choice for the top-10% campaign? Justify with the confusion-matrix trade-off.
Read the worked answer
Accuracy is not usable here: with 7% churn, a no-skill model that predicts 'no churn' for everyone already scores 93%. The reported 0.931 accuracy is essentially that baseline, and AUC 0.58 confirms the ranking is barely better than random. Switching to F1 at a 0.5 threshold is also a weak fix: F1 assumes precision and recall have equal business cost and does not encode the campaign capacity. For a top-10% email campaign, rank customers by predicted risk and evaluate precision at 10% (or recall at a 10% contact rate) and lift over the random 10% baseline. The confusion matrix should be read at the operating threshold that selects 10% of the test set, not at 0.5. Lowering the threshold to 0.2 would increase recall but likely flood the campaign with low-precision contacts; without a good ranking, no threshold fixes the model. Use the lift chart and precision-recall curve, then check calibration before setting the final cutoff.
2. What is wrong with the feature set and split, and how would you correct the experiment?
Read the worked answer
The cancel_reason field is leakage: it exists only after cancellation, so it cannot be known at scoring time and will inflate test performance. Remove it and audit support_tickets and monthly_spend for whether their values are captured before the prediction point. The random stratified split is acceptable for an i.i.d. estimate, but churn deployment predicts future customers, so a time-based split (train on older cohorts, test on the most recent) is more honest. In Dataiku, set the split to time-based with a date column or use a separate holdout. Retrain after removing leaky fields, and compare AUC, precision@10%, lift, and calibration. If a feature's importance collapses when leakage is removed, that confirms the issue. Also check that stratified splitting preserves the 7% churn rate in both sets.
3. The retrained model has AUC 0.79 and recall 0.55 at a 0.35 threshold. How would you decide whether to deploy?
Read the worked answer
AUC 0.79 is a ranking improvement, but deployment depends on the campaign's capacity and cost. At the 10% contact budget, compute precision@10%, lift, and the expected number of true churners reached; recall 0.55 at 0.35 is irrelevant if 0.35 selects far more than 10% of customers. Check the confusion matrix, calibration, and segment-level performance (tenure, plan, region). Compare against the current process and a no-model baseline. Run a small holdout campaign or shadow deployment to measure incremental saves and contact cost. Before launch, verify feature availability at scoring time, monitor drift, and set a retraining scenario. Deploy only if the incremental business value is positive and performance is stable across segments.