Churn Model Degradation After a Billing Migration and Delayed Labels
A B2B SaaS company deployed a churn propensity model nine months ago. At launch, offline AUC was 0.82; the latest out-of-time validation shows 0.74. The customer success team uses scores above 0.60 to trigger retention offers for accounts renewing within 90 days. The billing system was migrated in March and now emits plan names with 18% unseen categories. The monthly active users feature is computed from event logs that arrive up to three days late, and at scoring time it is often imputed with the last known value. Churn labels are observed only after the 90-day renewal window. The model has not been retrained since launch. The head of customer success reports that offers are increasingly hitting healthy accounts, while some large accounts churn without an alert.
1. Diagnose the likely sources of performance degradation and classify them as data drift, concept drift, or pipeline issue.
Read the worked answer
Three mechanisms are entangled. The billing migration creates data drift in a categorical feature: 18% unseen plan names mean the encoder either maps them to unknown or breaks assumptions, and plan tier is likely predictive. The delayed monthly active users feature is a pipeline and freshness issue: imputing with the last known value at scoring time makes training and serving distributions differ, so the model sees stale behavior for exactly the accounts being scored. A shift in churn drivers or base rates would be concept drift, but it should not be assumed until the first two are fixed. The 90-day label delay explains why the degradation was invisible for months. The reported drop from 0.82 to 0.74 is therefore not yet evidence that the relationship changed; it is at least partly a broken feature contract. Fix the plan mapping and feature freshness, then re-evaluate before retraining.
2. Design a monitoring and retraining plan that works with 90-day delayed labels.
Read the worked answer
With labels 90 days late, monitor leading indicators. Track score distribution and alert volume weekly; a rising share of scores above 0.60 with stable churn suggests over-flagging. Track feature drift with PSI or KS for numeric features and unseen-category rates for categoricals, especially plan tier and monthly active users age. Monitor pipeline freshness: percentage of scores using a monthly active users value older than one day, null rates, and join failures. Use early proxies such as login decline, support ticket sentiment, and payment failures as leading signals, but validate them against eventual churn. Backfill labels when the 90-day window closes and compute a rolling out-of-time AUC and calibration by cohort. Retrain on a fixed cadence, such as quarterly, and trigger an off-cycle retrain when drift or proxy degradation exceeds thresholds. Deploy challenger models in shadow mode before promotion.
3. What handoff controls would you put in place with customer success and engineering?
Read the worked answer
Create a feature contract with billing and event-log owners: allowed plan values, refresh SLA, late-arrival handling, and a defined unknown category. At scoring time, if monthly active users is stale beyond the SLA, either use a model variant without that feature or mark the score as degraded rather than silently imputing. Add a fallback score and a clear reason code. Document the model card, training window, label definition, and known limitations.
Give customer success a runbook: what a score means, which accounts to prioritize, how to record offer outcomes, and how to report a suspicious score. Engineering needs versioned model artifacts, reproducible feature snapshots, and a rollback path. Set an alert when unseen plan categories exceed 1% or feature freshness misses SLA. Finally, define who owns retraining and who approves promotion, so the model does not sit unchanged for another nine months.