The Churn Model With a Leaky Feature
A subscription fitness app builds a churn model. The data science team reports AUC 0.97 on a held-out test set and the marketing VP wants to ship it to prioritize retention offers. Features include tenure, app sessions per week, support contacts, plan tier, and 'days since last payment failure resolved'. The test set was created by a random 80/20 split of the last 18 months of subscriber records. When the team scores current subscribers, the model flags a suspiciously large share of long-tenured, high-engagement users as high churn risk. A data engineer mentions that the payment-failure field is updated by a nightly job that also marks the account 'churned' in the same batch when a cancellation is processed.
1. Diagnose the most likely cause of the 0.97 AUC and the odd production behavior.
Read the worked answer
Target leakage plus a temporal split problem.
Leakage: 'days since last payment failure resolved' is written by the same nightly batch that marks accounts churned. For churned accounts, this field can encode post-cancellation state - e.g., it goes null or freezes at a value correlated with the cancellation event. The model learns 'this field looks like X → churned' without any causal or predictive pathway available at scoring time. That inflates AUC toward 0.97.
Temporal split: a random 80/20 split over 18 months lets the model train on records from after the test records. Subscriber behavior drifts, and leakage features are especially dangerous under random splits because the same batch job wrote both train and test rows.
Production symptom: current subscribers have not churned, so the leaky field behaves differently than in training. The model, deprived of its crutch, falls back on whatever weak correlations remain - producing the implausible pattern of flagging engaged long-tenured users.
AUC near 0.97 on a churn problem with modest features is itself a red flag worth investigating before shipping.
2. Describe the validation plan you would require before this model is used for retention offers.
Read the worked answer
1. Feature audit for leakage. For each feature, ask: is it knowable at the scoring moment, and is it written by any process that also observes the outcome? Drop or re-derive 'days since last payment failure resolved' with an as-of timestamp. Audit the nightly job's write path.
2. Time-based split. Train on months 1–12, validate on 13–15, test on 16–18. No random shuffling across time. Confirm features are computed as-of the prediction date (point-in-time correctness).
3. Realistic baseline. Compare against a simple tenure + engagement rule. If the model can't beat it meaningfully, the complexity isn't earning its keep.
4. Metrics beyond AUC. Precision at the top-K the retention team can actually act on (e.g., top 5,000 offers/month), calibration, and lift over baseline. AUC is threshold-free and hides operational tradeoffs.
5. Shadow mode. Score live traffic for 4–6 weeks without acting, compare predicted risk to observed churn, then decide on rollout.
6. Monitoring plan. Track feature drift, especially the payment field, and score distribution shift.