Predictive Scores That Look Great in Training
A marketing team builds a churn model in Alteryx on 18 months of customer data (~90,000 customers). They split 70/30 into train/validation, fit a logistic regression, and see 94% accuracy on validation. When they score the current customer base and hand the top-decile list to the retention team, the team reports that almost none of the "high-risk" customers actually churned in the following quarter-the list looks random. The data includes a last_login_days_ago field and a cancelled_date field. The model's top predictors are last_login_days_ago and support_tickets.
1. Give the most likely reason the model's validation accuracy did not translate to real-world performance, and how you would confirm it.
Read the worked answer
The red flag is the cancelled_date field. If it is present in the feature set, the model is leaking the outcome: a customer who cancelled has a populated cancelled_date, and that field is essentially a copy of the target. The model learns "cancelled_date is not null → churned" and scores 94% in validation, because validation is drawn from the same historical data where cancellation already happened. But when scoring the current base-customers who have not yet cancelled-that field is null for everyone, so the learned signal is useless and the rankings are effectively noise. This is target leakage, and it explains the train/validation gap versus live performance.
Confirm it: (1) Inspect the feature list and remove any field derived from or recorded at/after the outcome (cancelled_date, cancellation reason, refund flags, post-churn activity). (2) Re-run and compare-accuracy will drop to a realistic level (often 70–85% for churn). (3) Check whether any single feature has implausibly high importance or near-perfect univariate separation. (4) Verify the validation split is time-based: if train and validation are random rows from the same 18 months, the model can also leak via future information. Use a time-based holdout (train on months 1–12, validate on 13–18) so validation mimics the real forecasting task.
Secondary suspect: last_login_days_ago may be measured after churn for churned customers, another leak. Confirm the field's as-of date precedes the prediction point.
2. The team rebuilds without leakage and gets 78% accuracy but still worries the top-decile list is weak. What evaluation approach would you recommend instead of accuracy, and why?
Read the worked answer
Accuracy is the wrong metric for a top-decile retention list because churn is imbalanced and the business only acts on a small slice. If, say, 8% of customers churn, a model that predicts "nobody churns" scores 92% accuracy and is worthless. The team cares about who is at the top of the ranking, not overall correctness.
Recommend:
- Rank-based metrics: precision at the top decile (of the 10% flagged highest-risk, what fraction actually churned?) and lift over baseline. Lift = (churn rate in top decile) / (overall churn rate). A useful list should show meaningful lift; random ranking gives lift ≈ 1.0, which matches the "looks random" complaint.
- ROC AUC for overall ranking quality, and a precision-recall / cumulative gains curve because PR is more informative under class imbalance.
- A time-based validation set (score months 13–18, evaluate against actual churn in the following quarter) so the metric reflects the real deployment.
- Business-aligned threshold: pick the cutoff by how many customers the retention team can actually contact, then measure precision at that capacity, not at an arbitrary 0.5 probability cutoff.
Finally, compare against a trivial baseline (e.g., rank by last_login_days_ago alone). If the model doesn't beat the simple rule on top-decile precision, the complexity isn't earning its keep.