Churn Model with Target Leakage and Class Imbalance
A telecom operator has 120,000 :Customer nodes in Neo4j 5 and GDS 2.5. Only 8.1% churned in the last 12 months. The team builds a node classification pipeline with 42 numeric node properties, including tenure, monthlyCharges, supportTickets, avgDownloadGB, and riskScore, an upstream model output that was trained on the churn label. They add a degree node property, select all features, configureSplit with testFraction 0.2 and validationFraction 0.2, add logistic regression, and train. Test AUC is 0.994 and validation AUC is 0.991. When scoring 10,000 new customers, AUC falls to 0.63; riskScore is null for 37% of them and the observed churn rate is 1.9%. The business wants a reliable 30-day churn score and a defensible evaluation.
1. Diagnose why offline metrics are near-perfect but deployment performance collapses.
Read the worked answer
The near-perfect AUC is not credible for an 8.1% churn problem. riskScore is the obvious target leak: it was produced by a model trained on the churn label, and it is missing for 37% of new customers. The pipeline also uses a random split over one snapshot, so validation and test customers are drawn from the same period and the same feature distribution. That does not prove the model will work on next month's customers.
Remove riskScore from selectFeatures and retrain. Audit every feature for availability at scoring time and for derivation from the label. Replace the random split with a temporal design: train on older customers, validate and test on a later period, and keep a final holdout that is never used for tuning. Graph features such as degree must be computed only from data available before the prediction cutoff. Re-evaluate on the temporal holdout, not on the original random test split.
2. After removing riskScore, test AUC is 0.78 but recall for churners at threshold 0.5 is only 0.31. How should the team improve and report the model?
Read the worked answer
AUC measures ranking, not the operating threshold, and 0.5 is arbitrary for an imbalanced class. Report precision, recall, F1, and the confusion matrix for the churn class, plus precision-recall AUC. Choose a threshold from business costs: missed churners versus wasted retention offers. If the classifier configuration supports class weights, use them; otherwise compare logistic regression with a random forest and consider resampling. Do not optimize accuracy, which would predict 'no churn' almost always.
Add graph features that are legitimately available at scoring time, such as support-ticket network degree, community membership, or usage-decline patterns computed before the cutoff. Avoid features derived from neighbours' future churn labels. Use the pipeline's validation split for tuning and the temporal holdout for the final report. State the chosen threshold, the expected precision and recall, and the churn rate in the scoring population so the business can judge lift over baseline.
3. The team wants to add 'number of churned neighbours' as a feature. Should they?
Read the worked answer
Not if it is computed from the full graph using the current churn label. At scoring time the future churn status of a customer's neighbours is unknown, and the node classification pipeline computes node property steps before the train/test split. Such a feature leaks label information into training and will look strong offline and fail in production. The same problem applies to any feature built from post-cutoff edges or post-cutoff labels.
A historical version can be valid: count neighbours who churned before the prediction cutoff, using only relationships and labels known at that time, and make it available as a precomputed node property. Document the cutoff and refresh it consistently. Validate by checking that the feature has the same distribution and availability for new customers. If the team cannot guarantee that, exclude it rather than accept an inflated AUC.