Retention Model Drift After a Price Increase: Retrain, Recalibrate, or Re-Threshold?
An auto insurer's retention model predicts 90-day churn. It was trained 8 months ago on 1.4 million customers, with validation AUC 0.84 and precision 0.61 at the chosen 0.50 threshold. Since a March price increase, monthly PSI is 0.28 on tenure and 0.19 on prior claims; precision at the same threshold has fallen to 0.47, while AUC is 0.81. Marketing sends a $40 retention offer to everyone above 0.50, and offer volume has risen 38% with no lift in saves. The model owner wants to retrain immediately on the last 60 days of data; the marketing lead wants to lower the threshold to catch more churners. Label maturity is 90 days, so recent outcomes are incomplete. The deployment has no challenger model and no calibration monitoring. Average customer margin is $310 per year, and the retention team has a fixed monthly offer budget.
1. Should the team retrain immediately, recalibrate, or lower the threshold? Recommend a sequence and justify it.
Read the worked answer
AUC fell only 3 points while precision at a fixed threshold fell 14 points, which suggests calibration shift and a changed base rate more than a total loss of ranking power. The price increase likely changed the relationship between score and churn. Retraining on 60 days is risky because labels mature in 90 days, so recent data is censored and will understate churn. First, diagnose: compare score distributions, calibration curves, and precision by tenure and claims segment; check whether the price increase affected segments differently. Then recalibrate the existing model on mature data using isotonic or Platt scaling and re-evaluate. If ranking within segments has degraded, retrain with recent mature data plus sample weights, not raw 60-day labels. Run a challenger in shadow. Lowering the threshold is not a fix; it increases offer volume and cost without addressing score meaning. Decide with a cost matrix. The sequence is diagnose, recalibrate, then retrain if needed.
2. How should the operating threshold be set with a $40 offer, $310 annual margin, and a fixed budget?
Read the worked answer
Set the threshold from expected value, not from 0.50. If save probability conditional on accepting the offer is 0.35, expected value at churn probability p is p times 0.35 times 310 minus 40. Break-even is about p = 0.37. If save probability is lower, the threshold rises. Choose the cutoff that maximizes expected net benefit and then constrain by the fixed budget; rank by expected value per dollar. With precision 0.47 at the current threshold and rising volume, a lower threshold likely has negative marginal value. Segment thresholds can be justified only if they reflect differential save rates or margins and are fair-lending reviewed. Monitor precision at the operating threshold and calibration, not AUC alone.
3. What monitoring would catch this kind of degradation earlier, given delayed labels?
Read the worked answer
Add calibration monitoring using Brier score and predicted-versus-observed churn by score band, precision at the operating threshold, score PSI, and segment-level drift. Because labels are delayed, use leading indicators: offer acceptance, cancellation requests, and early tenure behavior. Set alert thresholds based on historical variation, for example calibration slope outside 0.9-1.1 for two consecutive weeks or precision at threshold down 5 points. Add a challenger pipeline and a retraining trigger tied to drift plus label maturity. Report segment-level results so a tenure-specific price effect is visible. The key is to separate ranking drift from calibration drift; they require different responses.