The Churn Model That Beat the Benchmark and Lost Money

The offline AUC is not evidence of campaign value, and several failures compound. The clearest defect is label leakage: features include usage and payment status from the same 90-day window as the churn label, so the model partly learns the outcome. Excluding payment-plan customers creates selection bias that makes the deployed population differ from training. But leakage alone does not explain a targeted group churning more than holdout; that suggests the campaign or its measurement is at fault. A matched holdout can be contaminated if targeting or contact spills over, and a $45 incentive can attract customers who were likely to leave anyway or signal that leaving is rewarded. The 14% versus 18% save rate among responders is conditional on response, which is not the same as incremental save. AUC ranks risk, but the campaign needs uplift: who changes behavior because of the offer. Finally, an 11-month-old model plus a new unlimited plan is severe drift. Treat the result as a decision-quality failure, not only a model-quality failure.

Redesign validation around the decision. Build a point-in-time feature store so every feature has an as-of timestamp strictly before the label window; enforce this with automated tests that fail if feature dates overlap the outcome period. Use time-based splits by snapshot month, not random splits, and validate on a later period that includes the new plan if available. Include payment-plan customers and report performance by segment, including calibration, not only AUC. Estimate incremental impact with a randomized or well-matched holdout and measure save rate as intent-to-treat, not responder-only. Consider an uplift model with a two-model or causal-forest approach and evaluate by expected net value at the chosen threshold. Add a business metric: margin saved minus incentive cost per contact. Reconcile the offline metric with campaign economics before any relaunch. The judgment is that a lower-AUC uplift model can be worth more than a high-AUC risk model.

Pause broad deployment and run a controlled shadow or small randomized pilot. Relaunch only after leakage is fixed, the model is retrained on recent data including the new plan, and holdout-based uplift is positive after incentive cost. Set guardrails: maximum contact volume, no offer to customers already flagged as high-risk involuntary churn, and a stop rule if treated churn exceeds control by a defined margin. Monitor input drift, score distribution, calibration, and campaign lift weekly; retrain on a schedule tied to plan changes and label maturity. Define an accountable business owner for the campaign, separate from the modeling team, who accepts the economic risk. Keep a permanent randomized control group so future results are interpretable. The judgment is that a model with strong offline discrimination but no proven incremental effect should not receive a 120,000-customer budget; the cost of a negative campaign is both money and organizational trust.