A Churn Model with a Suspicious 96% Accuracy
FitLoop is a subscription fitness app with 220,000 active customers and a monthly churn rate of 4.1%. A junior data scientist has built a churn model in Python with scikit-learn. The script reads a customer export, drops rows with missing values, one-hot encodes categorical fields, standardises every numeric field, splits the data randomly 80/20, trains a gradient-boosting classifier and reports 96.2% accuracy and 0.94 ROC-AUC. The feature list includes days since last login, support tickets in 30 days, monthly minutes, discount percent, account status, last payment date and cancellation reason. The plan is to deploy the model next sprint. Product managers are excited because the score looks far better than the existing 61% rule-based hit rate. The data scientist has not documented random seeds, package versions or the exact export date. A colleague notices that account status is recorded after cancellation and that cancellation reason is only populated for customers who have already left.
1. Diagnose why 96.2% accuracy and 0.94 ROC-AUC are not credible. Identify at least three concrete leakage or evaluation errors.
Read the worked answer
Three errors stand out. First, account_status and cancellation_reason are post-outcome variables: they are known only after churn, so the model is effectively told the answer. Last_payment_date may also encode imminent cancellation. Second, 96.2% accuracy is barely above the 95.9% majority-class baseline (100% minus the 4.1% churn rate); accuracy hides failure on the minority class. Third, a random 80/20 split on customer data is optimistic for a time-dependent process: future customers and future behaviour leak into training, and the same customer may appear in both sets if the export has multiple rows per customer. Standardising before the split also leaks test-set statistics, though the effect is smaller. Dropping rows with missing values can bias the sample if missingness is informative. The 0.94 ROC-AUC is therefore not a trustworthy estimate. The existing 61% rule-based hit rate may be measured differently, so compare on the same period, definition and threshold.
2. Rewrite the modelling pipeline at a design level so that the reported estimate would be trustworthy. Include split, preprocessing, metrics and reproducibility controls.
Read the worked answer
Split by time, not randomly: train on the oldest 70%, validate on the next 15%, test on the most recent 15%, and group by customer ID. Drop account_status, cancellation_reason and any field recorded after the prediction point; document the prediction timestamp. Put imputation, encoding and scaling inside a scikit-learn Pipeline fitted only on training folds. Use stratified or time-series cross-validation for tuning. Report PR-AUC, recall at a fixed precision, calibration and the confusion matrix at the proposed threshold, not accuracy alone. Establish a majority-class baseline and the existing rule baseline. For reproducibility, pin package versions, set and record random seeds, version the export and feature definitions, and save the fitted pipeline with a data schema check. Add unit tests for leakage, such as asserting that forbidden columns are absent, and a backtest on the last three months. Only then compare with the 61% rule.
3. The product team wants to deploy next sprint. What would you tell them, and what minimum evidence would you require before launch?
Read the worked answer
I would not deploy next sprint. The current evidence is a leakage-inflated offline score, not a validated business case. Minimum evidence: a time-based test result on recent data with post-outcome fields removed, calibration, and a threshold chosen against the retention team's capacity, for example how many customers can be contacted per week. The model must beat both the majority baseline and the rule-based approach on the same metric, such as lift in churners captured at the contact budget. Run a small holdout pilot with a control group, because a churn model changes outcomes only if the intervention works. Add monitoring for input drift, score distribution, calibration and business lift; define rollback triggers. Also check fairness across customer segments. If the corrected model is only marginally better than rules, a simpler rule may be the better launch.