Churn Model Monitoring When Labels Arrive Late
A subscription analytics company deployed a churn propensity model for its customer success team 11 months ago. At launch, holdout ROC-AUC was 0.82 and calibration error was 0.03. Monthly scoring covers 85,000 accounts. The team contacts accounts with predicted risk above 0.65; about 1,100 accounts are contacted per month. Last quarter, ROC-AUC fell to 0.74 and the contacted group's actual churn rose from 19% to 27%. Input monitoring shows PSI 0.31 for tenure, 0.18 for support tickets, and 0.05 for plan type. The label is available only after a 90-day lag. Marketing launched a new discount campaign and changed the onboarding journey eight weeks ago. The model retrains quarterly on a rolling 24-month window; the last retrain was four months ago.
1. Is this data drift, concept drift, or a measurement problem? Explain what the evidence supports and what it does not.
Read the worked answer
The evidence supports covariate shift plus a possible concept or selection change, but it does not yet prove model degradation. PSI of 0.31 for tenure and 0.18 for support tickets indicates the input distribution has moved materially; plan type is stable. A drop in ROC-AUC from 0.82 to 0.74 is consistent with that shift, but the label lag matters: recent outcomes are incomplete, and the AUC may be computed on a biased matured cohort.
The rise in actual churn among contacted accounts, from 19% to 27%, is not direct evidence of worse ranking. Those accounts are selected by the model's threshold, so a changed score distribution can change who is contacted and therefore the observed churn rate. The marketing campaign and onboarding change also altered the base churn process. That is concept drift or intervention effect, not just input drift.
Separate the diagnoses: check score distribution and calibration on matured cohorts, performance by tenure and ticket segments, and whether the contact threshold now selects a different mix. Do not retrain immediately on recent labels; with a 90-day lag and an eight-week-old campaign, the training data are incomplete and confounded.
2. Design a monitoring plan that works with a 90-day label delay. Include leading indicators, matured-cohort evaluation, and retraining triggers.
Read the worked answer
Use two layers. Leading indicators, available immediately: score distribution, PSI/KS on top features, missingness, override rate by the success team, contact volume, early cancellation signals, login frequency, support escalations, and discount uptake. These detect input and process change but not accuracy.
Matured-cohort evaluation, available after 90 days: calibration error, ROC-AUC and PR-AUC, recall in the top contact decile, and observed churn versus predicted risk for each monthly cohort. Always evaluate on fully matured cohorts and report sample sizes. Reconcile the label definition because the campaign may change what cancellation means.
Set triggers in advance: calibration error above 0.05 on a matured cohort; top-decile recall drop greater than 10% relative to launch; PSI above 0.2 on two consecutive months for a top feature; or business KPI degradation such as contact-to-save rate falling. Retraining should be triggered by evidence and validated out of time, not just because the quarter ended. A monthly retrain on a rolling 24-month window may be reasonable, but only with a frozen holdout and a canary.
3. How should the discount campaign and onboarding change affect the response? What would you recommend to the customer success team now?
Read the worked answer
Treat the campaign and onboarding change as interventions that can change both the features and the churn process. A discount campaign may retain accounts that would otherwise churn, lowering the observed churn rate, or attract a different mix of accounts, raising it. The onboarding change may alter early tenure behavior, which is exactly the feature showing the largest drift.
The current model's scores are not directly comparable across the pre- and post-campaign periods. Recommend: keep the existing model in production but lower confidence in the 0.65 threshold; add a manual review or a secondary rule for high-tenure accounts; and run a randomized or holdout comparison of the campaign so its effect can be estimated separately from model performance. Add campaign exposure and onboarding cohort as features or as segments, and evaluate whether a separate post-campaign model is needed.
Do not retrain on the last 24 months without marking the intervention period. Instead, build a versioned dataset with campaign flags, wait for matured labels, and compare the challenger on post-campaign cohorts. If contact capacity is limited, prioritize by expected saved revenue rather than score alone until calibration is restored.