The Recommendation Engine That Learned the Wrong Lesson
A retail team has trained a binary classifier on 18 months of purchase data to predict whether a customer will buy a premium subscription. The dataset contains 1,000,000 rows; only 2% of customers ever subscribed. Before deployment, the model reports 96% accuracy on a held-out test set. The marketing team is enthusiastic and plans to target everyone the model labels positive. A data scientist notices the model predicts negative for almost every row, including customers who actually subscribed, and that the training data includes a 'campaign_exposure' column recorded after each subscription decision. The team has one week before launch and no budget for new data collection.
1. Diagnose why 96% accuracy is a misleading signal here, and explain what evaluation approach the team should use instead.
Read the worked answer
The dataset is heavily imbalanced: only 2% of rows are positive. A model that predicts negative for every single row would still be correct on roughly 98% of cases, so 96% accuracy is below the trivial baseline and tells us nothing about whether the model finds subscribers. This is the classic accuracy paradox on skewed data.
The symptom the data scientist observed - nearly all predictions negative, including true subscribers - is consistent with a model that has collapsed to the majority class. Accuracy hides this because it averages over the 98% negatives.
The team should switch to metrics that are sensitive to the minority class: precision, recall, F1, and especially the confusion matrix broken out by class. For a targeting decision, recall matters most if missing a likely subscriber is costly, while precision matters if outreach is expensive. A precision-recall curve and its area are more informative than an ROC curve under this level of imbalance, because ROC can look deceptively good when negatives dominate.
They should also compare against a naive baseline (always predict negative) and a simple rule, and consider resampling or class weighting during training. Finally, they should evaluate on a time-based split rather than a random split, since the deployment target is future customers.
2. The 'campaign_exposure' column is flagged as a problem. Explain the issue and what the team should do about it before launch.
Read the worked answer
'campaign_exposure' is recorded after the subscription decision, meaning it is a post-outcome variable. If exposure only happens because a customer subscribed, or is logged during the same campaign that produced the label, the model can learn to key on a feature that will not be available, or will not mean the same thing, at inference time. This is target leakage: the feature encodes information about the label that the model would not legitimately have when scoring a new customer.
Leakage inflates offline metrics and produces a model that looks strong in testing and fails in production, which matches the gap between the reported 96% accuracy and the observed behavior.
The team should audit every feature's timestamp relative to the label timestamp and drop or reconstruct any feature that is generated at or after the outcome. For campaign_exposure specifically, they should confirm when it is written and whether it is knowable before the prediction point. If it is not, remove it and retrain, then re-evaluate with precision and recall. They should also document a feature-availability rule so future features are screened the same way, and validate the retrained model on a time-based holdout that mimics the launch population.