The Churn Model That Passed Validation and Failed in Production

Diagnosis - four independent failures, each sufficient to explain the outcome:

1. Temporal leakage / concept drift. Training data predates the pricing change and new cancellation flow. The relationship between features and churn has changed; the model learned a world that no longer exists. AUC on old data says nothing about current behavior.
2. Random split leakage. With a random split, the same customer appears in train and test. The model memorizes customer-specific patterns and validation overstates generalization. This alone can inflate AUC substantially.
3. Horizon mismatch. The model predicts 30-day cancellation; retention acts on 90 days. A model optimized for the wrong target will rank the wrong customers regardless of accuracy. This is a requirements failure, not a modeling failure.
4. Feature drift. The top feature (support tickets) was structurally reduced by the new flow, so the model's strongest signal has lost variance and its learned weight is now misleading.

How validation should have been designed:

• Out-of-time validation: train on earlier periods, validate on the most recent period, never random-split when the deployment is forward in time. Use time-based or customer-grouped splits to prevent entity leakage.
• Target aligned to the decision: define the label as the 90-day outcome retention acts on, and validate against the retention team's actual intervention economics (who is worth contacting), not raw AUC.
• Drift monitoring: track feature distributions and label rates in production against training; alert on shift. The support-ticket collapse would have triggered an alert.
• Stability checks: evaluate performance on the post-change period specifically before deployment.

Immediate actions:

1. Stop using the model for targeting but do not delete it; retain it for audit. Revert to a documented heuristic or holdout-based prioritization in the interim.
2. Rebuild the label to the 90-day horizon and retrain on post-change data, validating out-of-time on the most recent period. If post-change history is too short, state that explicitly and treat any model as provisional.
3. Re-examine the feature set: drop or re-derive support-ticket features that the new flow structurally altered; add features reflecting the new cancellation flow.
4. Evaluate against the decision, not AUC: measure lift in the top decile the retention team can actually contact, and expected retained revenue per contact.
5. Confirm the model owner gap is filled with named accountability before redeployment.

Governance changes to prevent recurrence:

• Model risk tiering: any model driving customer-facing or revenue decisions requires independent validation with out-of-time testing and documented target rationale.
• Deployment gate: require evidence of temporal validation, drift monitoring, and a named owner with a documented handover. The owner's departure should have triggered a review, not a silent orphan.
• Change-triggered review: material business changes (pricing, cancellation flow) must trigger a model impact assessment. The pricing change was known and should have flagged the model.
• Monitoring in production: automated drift alerts on top features and outcome rates, with a defined response owner.

Root cause framing for leadership: the failure was not the algorithm; it was the absence of a lifecycle process. The 0.86 AUC was a measurement artifact, and the organization had no mechanism to notice when the world moved.