Churn Model Drift After a Pricing Change
A subscription retailer launched a new pricing tier in March. Its churn classifier, trained on 18 months of pre-March data, had an AUC of 0.81 in February. By May the AUC on monthly holdout data fell to 0.68, though the model itself was untouched. The team also notices that the predicted churn rate averaged 6% in February but 14% in May, while actual churn rose only from 5% to 8%. The new tier is over-represented among customers flagged as high risk, and the discount variable used in training now has a different distribution because the tier bundles a permanent discount. Leadership asks whether to retrain immediately, recalibrate thresholds, or roll back the model. The data team has only 10 weeks of post-launch labeled churn data, with churn labels arriving about 60 days after the decision point.
1. Diagnose the most likely cause of the AUC drop and the gap between predicted and actual churn. What evidence in the setup supports your diagnosis?
Read the worked answer
The pattern points to covariate shift plus label lag rather than a broken model. The AUC drop means the ranking of risk became less separable; the predicted-versus-actual gap (14% predicted vs 8% actual) is a calibration failure layered on top. The discount variable changed distribution because the new tier bundles a permanent discount, so the model sees values it rarely saw in training and extrapolates. The new tier is over-represented among high-risk flags, meaning the model is partly reacting to a population it never learned. The 60-day label lag means May holdout labels are incomplete: recent cohorts are censored, so observed churn understates true churn and biases both AUC and calibration. Before concluding the model degraded, the team should check label maturity, compare feature distributions pre- and post-launch, and evaluate AUC only on fully matured cohorts.
2. Given only 10 weeks of post-launch labels, recommend a course of action and justify the trade-offs among retraining, recalibration, and rollback.
Read the worked answer
Rollback is weak because the pre-March model no longer matches the customer base and the pricing change is permanent. Immediate full retraining is also risky: 10 weeks of labels with a 60-day lag gives roughly one matured cohort, too little to learn stable weights and prone to overfitting the new tier. The best sequence is: (1) restrict evaluation to matured cohorts to get an honest baseline; (2) apply post-hoc calibration (for example isotonic or Platt scaling) on matured post-launch data to close the 14%-vs-8% gap without changing ranking; (3) adjust the decision threshold to reflect the new base rate and the business cost of false negatives versus false positives; (4) add the tier and discount interaction as explicit features and monitor drift weekly; (5) schedule full retraining once at least three matured monthly cohorts exist. This buys calibration accuracy now while avoiding a model trained on censored labels.
3. What monitoring would you put in place to catch this earlier next time, and what would trigger escalation?
Read the worked answer
Monitor three layers. Input drift: population stability index or KS tests on each feature weekly, with special attention to discount and tier, alerting when PSI exceeds about 0.2. Output drift: track mean predicted probability against the matured actual rate on a rolling basis, and watch the calibration curve, not just AUC. Performance drift: compute AUC and calibration only on cohorts whose 60-day labels have matured, and display label completeness alongside every metric so censoring is never mistaken for degradation. Escalation triggers: PSI above threshold on a top feature, a calibration gap exceeding a set tolerance (for example 3 percentage points) for two consecutive matured cohorts, or any new category share above 10% that was absent in training. Tie each trigger to an owner and a decision, such as recalibrate, reweight, or retrain.