The 96.8% Churn Model

96.8% accuracy is below the do-nothing baseline.

• Base rate is 3% churn. A model that predicts 'nobody ever churns' scores 97.0% accuracy and produces zero useful output. The model is worse than useless on this metric.
• Accuracy is dominated by the majority class; it averages away the only cases anyone cares about.

What to measure, driven by the constraint:

• The team can make 2,000 calls/month. That is a capacity problem, so the right metric is precision at the top 2,000 ranked accounts - of the 2,000 people we call, how many would actually have churned?
• Also report recall (share of all churners reached) and lift at the cutoff versus calling 2,000 at random.
• AUPRC is far more informative than AUROC under 3% prevalence; AUROC looks flattering because the true-negative pool is enormous.
• Turn it into money: expected value per call ≈ P(churn) × $520 saved − $40 offer cost. Break-even is P(churn) ≈ 7.7%, so the call list should be truncated where predicted probability crosses ~0.08, not at an arbitrary 2,000. If the model ranks fewer than 2,000 accounts above break-even, calling the rest destroys value.
• A confusion matrix at the chosen threshold makes the trade-off explicit for the business owner.

Target leakage in the feature set.

• account_closed_date is the label in disguise. A churned account has a closed date; an active one does not. The model learns 'closed date is not null' - perfect training performance, zero predictive value in production, because at scoring time you only score accounts that are still open.
• cancellation_reason is captured by an agent after the customer calls to cancel. It is an outcome, not a predictor. Including it means the model only 'predicts' customers who have already decided.
• Both must be removed from the feature set entirely, not just down-weighted.

Temporal leakage in the split.

• A random 70/30 split across all months scatters the same customer's adjacent months into both train and test. The model memorizes individuals rather than learning a generalizable pattern, and it trains on the future to predict the past.
• Remedy: split by time - train on months 1–18, validate on 19–21, test on 22–24 - or group-split by customer if you must pool months. Add a purge gap between periods so a window straddling the boundary doesn't leak.

After fixing both, expect accuracy to drop sharply. That drop is the honest number; a leakage-free model at 80% accuracy with useful lift is a real result.

Reject the plan as stated.

SMOTE before the split synthesizes minority points from the whole dataset, then splits. Synthetic points derived from test-set customers end up in training, and neighbours of test-set points leak across the boundary. Reported performance becomes optimistic and collapses in production. Classic leakage-by-preprocessing-order.

Correct sequence:

1. Split first (time-based, per the leakage fix).
2. Resample inside each cross-validation fold on training folds only - never touch validation or test.
3. Apply the identical transformation to scoring-time data - which, for SMOTE, means none: production data is not resampled.

Alternatives that are often better here:

• class_weight='balanced' or a scale factor. No synthetic records, no leakage risk, and it keeps predicted probabilities roughly calibrated - which matters because the decision is a probability threshold against a $40/$520 trade-off.
• Threshold tuning at the break-even probability rather than resampling. The ranking is what the call list needs; if ranking is decent, resampling changes little.
• If SMOTE is genuinely required, restrict it to SMOTE(sampling_strategy=0.1) inside a pipeline object so it cannot be applied out of order.

Validate against reality: backtest the top-2,000 list on the held-out final 3 months and confirm realized churn in the called group matches the predicted rate. If it doesn't, the pipeline still has leakage.