A 0.99 AUC That Should Not Ship
Marco's team built a churn classifier for a subscription business with 50,000 accounts; 4.2% of accounts churned during the six-month label window. The first model reported 96.1% accuracy and ROC-AUC 0.99 on a random 80/20 split. The feature table includes days_since_last_login, monthly_spend, support_tickets_90d, contract_end_date, and cancellation_reason. Numeric features were standardized across the whole dataset before splitting, then a random forest was fit with default settings. The extract spans January 2019 to December 2024 and the label is derived from cancellation_date. The marketing director wants to start emailing retention offers next week and has budget for 4,000 offers per month; each offer costs about $8, and the twelve-month margin on a retained account averages $120. Marco's manager says the metrics look 'too good' and asks for a review before anything ships.
1. Diagnose the implausible metrics. Which features and pipeline steps are producing leakage, and why does 96.1% accuracy tell you nothing here?
Read the worked answer
Accuracy is the wrong headline. With a 4.2% positive rate, predicting 'nobody churns' scores 95.8%. The model's 96.1% is roughly one percentage point better than a constant predictor that costs nothing to build. Accuracy is dominated by the negative class and cannot distinguish a useful model from a useless one at this base rate.
AUC 0.99 is a leak signal, not a triumph. Multiple leaks are plausible:
cancellation_reasonis populated at or after cancellation. It is an outcome, perfectly correlated with the label in the training data, and absent at scoring time for anyone still active. This alone can push AUC toward 1.0.contract_end_dateis often updated when an account cancels; if it reflects post-event state, it encodes the label.days_since_last_loginwas computed at extract time (December 2024) rather than at a fixed snapshot. For churners, the last login is by definition long before the extract, so the feature measures 'how long ago this account churned' - the answer, not the predictor.- Standardizing before the split fits the scaler on test rows, leaking distributional information. Minor for tree models, but it shows the pipeline has no enforced train/test boundary.
- A random split on data spanning 2019–2024 lets future behaviour inform predictions of the past, which no production deployment can reproduce.
Every feature must be answerable from data that existed at the prediction moment.
2. Redesign the split and evaluation so the reported numbers would survive contact with production. Name the metrics you would report and why.
Read the worked answer
Rebuild the feature table around a snapshot date. Pick a scoring moment (e.g., 1 January 2024), compute every feature using only events strictly before it, and define the label as churn within the following six months. Repeat across several rolling snapshots to get more than one training window. Drop cancellation_reason entirely and re-derive days_since_last_login as of the snapshot.
Split by time, not at random. Train on snapshots ending before the holdout period; validate on the most recent window. Use TimeSeriesSplit for tuning rather than shuffled K-fold. Wrap preprocessing in a Pipeline with a ColumnTransformer so the scaler is fit only on training folds.
Metrics that match the decision.
- PR-AUC (average precision) as the primary ranking metric - at 4.2% positives, ROC-AUC flatters models because the vast negative class inflates the false-positive rate denominator until it looks harmless.
- Precision and recall at the operating threshold, plus the confusion matrix in account counts, not percentages.
- Calibration curve if scores will be used to rank or to compare against cost. A random forest's votes are not probabilities, so check this explicitly.
- Lift over a no-model baseline (top 4,000 by score vs. 4,000 chosen at random) - the number the marketing director actually cares about.
Report a confidence interval on the holdout metric; a single point estimate from one split invites exactly the overconfidence that produced 0.99.
3. Marketing can send 4,000 offers per month. Using the stated economics, what threshold should the team use, and what else must be checked before promising the result?
Read the worked answer
Set the threshold from expected value, not convention. Offering costs $8; a retained account is worth $120 in margin. Contacting an account is worth it when p × $120 > $8, i.e. p > 0.067. So the operating threshold is about 0.067, not the 0.5 default that scikit-learn's predict() implies - and using 0.5 would suppress most of the genuinely at-risk accounts.
Then apply the capacity constraint. Rank all active accounts by calibrated score, take the top 4,000, and inspect the score of the lowest-ranked account admitted. If that score is comfortably above 0.067, budget is the binding constraint and the threshold is whatever the 4,000th score is; if it is below 0.067, expected value is negative for the tail and the campaign should send fewer than 4,000. Compare against sending to 4,000 accounts at random and state the incremental retained accounts.
Caveats to state before promising a number. The $120 assumes the offer actually works - the team needs a holdout group of at-risk accounts who receive nothing, otherwise there is no evidence that contact changed behaviour, and 'saved' accounts may have been low-risk anyway. Also check that the score is calibrated (a forest's raw vote is not a probability), that the 4.2% base rate in the training window matches current conditions, and that features are computable daily at scoring time. Without the holdout, the campaign's ROI is an assumption dressed as a forecast.