Feature Leakage in a Churn Pipeline
A subscription company builds a churn model. The pipeline computes, for each customer, 'days_since_last_login', 'support_tickets_30d', and 'avg_session_minutes_90d', then trains logistic regression with 5-fold cross-validation and reports AUC 0.93. A data scientist notices the feature-engineering step (imputation of missing session data plus standardization) is applied to the full dataset before the CV split. Separately, 'days_since_last_login' is computed as of the snapshot date, but churn is labeled as 'cancelled within 30 days after snapshot.' When the model is deployed, AUC on new customers drops to 0.71 and the team is puzzled because the offline CV was stable across folds.
1. Identify the leakage mechanisms present and explain how each inflates the offline estimate.
Read the worked answer
Two distinct leaks.
Preprocessing leakage: imputation and standardization fit on the full dataset leak information from validation folds into training. The leakage is usually small for standardization (means/variances are stable), but imputation using global statistics can leak target-correlated structure, especially if missingness correlates with churn. The fix is to fit all transforms inside each CV fold (a Pipeline), so validation data never informs the transform.
Temporal/label leakage: 'days_since_last_login' at snapshot is measured before the label window, which is legitimate if the snapshot truly precedes the label period. But if the snapshot date is not strictly before the 30-day window for every customer, or if session features are aggregated over a window that overlaps the churn window, the feature encodes post-outcome behavior. A customer who cancels often stops logging in, so low days-since-login correlates with churn only because it is measured after the decision. That is target leakage and can produce a large, fold-stable inflation - consistent with 0.93 vs 0.71.
The stability across folds is not evidence of validity; leakage that affects all folds equally is stable and wrong.
2. Propose a corrected validation design that would have caught this before deployment.
Read the worked answer
Use time-based validation that mirrors deployment. Define a strict cutoff: features computed only from data strictly before the snapshot, labels from the following 30 days, and no feature window overlapping the label window. Split by time (train on earlier cohorts, validate on later cohorts), never randomly, because random splits let future information into training.
Wrap all preprocessing in a pipeline fit only on the training portion of each split. Additionally, run a leakage audit: for each feature, check whether it could only be known after the outcome (e.g., cancellation-related events), and test a 'shuffled-label' sanity check - if AUC stays high with randomized labels, leakage is present. Finally, backtest on a held-out recent cohort to confirm the ~0.71 level is the honest estimate; the gap between 0.93 and 0.71 should be explained, not ignored.
3. The team wants to keep 'days_since_last_login' because it is the strongest feature. What is the defensible way to retain recency signal?
Read the worked answer
Retain recency only in a form that is strictly pre-snapshot and causally available at scoring time. Compute days-since-last-login as of the snapshot timestamp using only events before it, and verify the label window begins after the snapshot for every customer. If the deployment scoring happens at the same moment as the snapshot, this is valid and should be kept.
The problem arises only if the feature is computed at a time that overlaps the label period. The defensible approach: (a) document the exact timestamp semantics, (b) recompute features with a hard as-of join, (c) re-validate with time-based splits, and (d) confirm the feature's univariate AUC against the label is plausible and not near-perfect. If recency is genuinely predictive pre-snapshot, it will survive; if its power came from post-outcome measurement, it will collapse and should be dropped. Strength alone is not a justification to keep a leaky feature.