The Overfit Churn Model
A churn model is built in Alteryx with 60 predictors and ~18,000 labeled customers; churn rate is 4%. The team uses a single 70/30 split, reports 0.94 AUC on the holdout, and the business is about to deploy it. When a fresh quarter of data arrives, AUC drops to 0.61. The positive class is rare, and several predictors are highly correlated (account age, tenure, and days-since-signup). The team wants a model that will survive the next quarter and a defensible performance estimate.
1. Explain the 0.94-to-0.61 collapse and propose a more defensible validation design.
Read the worked answer
A single 70/30 split on a 4% positive rate gives a holdout with roughly 216 churners; AUC on so few positives is high-variance, and a lucky split plus leakage-prone or overfit predictors can inflate it. The collapse suggests the holdout was not representative and/or the model learned noise (60 predictors, correlated features, rare class).
Defensible design: k-fold cross-validation (e.g. 5-fold) stratified on the target so each fold preserves the ~4% rate, reporting mean and spread of AUC rather than one number. If data is time-ordered, use a temporal holdout (train on earlier quarters, test on the latest) because that mirrors deployment. Report precision/recall or lift at a chosen threshold in addition to AUC, since AUC hides the rare-class operating point.
2. What modeling and feature changes would you make given the correlation and rare-class issues?
Read the worked answer
Reduce collinearity: drop or combine tenure/account age/days-since-signup into one feature, and use a Field Info / correlation check to prune redundant predictors before modeling. With a rare class, set class weights or use a model that handles imbalance (e.g. logistic regression with balanced class weights, or a tree ensemble with class weighting) rather than resampling blindly.
Validate that the chosen threshold gives acceptable precision at the business's tolerable false-positive rate. Keep a simple, interpretable baseline (logistic regression) to compare against the ensemble; if the ensemble does not beat the baseline on cross-validated lift, prefer the baseline for stability. Re-check feature importance stability across folds.