AWS Machine Learning Specialty (AWS): Churn Model With 0.94 AUC and 0.04 Precision in Production

Three separate problems compound here.

1. Leakage in the split. A random row split puts the same customer in train and test in different months. The model learns customer identity and next month's outcome indirectly. Split by time instead (train months 1-18, validate 19-21, hold out 22-24) so the validation set asks the real question: predict a future month from the past.
2. Target leakage in a feature. account_closed_reason is only populated after the customer has closed, i.e. after the event being predicted. It must be dropped, not imputed. Check support_tickets_last_30d too: if the ticket extract is refreshed after the label window, it carries post-outcome information.
3. Metric mismatch. AUC measures ranking and is nearly insensitive to a 2.1% base rate, so 0.94 sounds better than it is. At that prevalence PR-AUC (0.62 was already the honest signal) and precision at the actual operating capacity are the metrics that govern the business outcome. A 0.5 threshold on a 2% base rate is arbitrary and produces exactly the 4% precision observed.

Do not resample or SMOTE before the split; use class weights or scale_pos_weight inside the model, evaluated on the untouched time-based holdout.

The constraint is capacity, not a probability threshold, so deploy a ranking with a fixed top-K, not a fixed cut-off.

1. On the time-based validation period, compute the model's ordering and measure precision@500 and recall@500 - the fraction of real churners captured by the top 500 accounts that week.
2. Compare against the do-nothing baseline: 500 random accounts capture about 500 x 2.1% ≈ 10.5 churners. If the model's top 500 captures, say, 60, the lift is roughly 5-6x and the expected value per call is (precision x save rate x account value) minus call cost. That calculation, not the AUC, is what justifies the project.
3. Deploy by sorting scored accounts and taking the top 500 per week. Recompute the effective threshold each week, because with a drifting base rate a fixed probability silently changes the number of calls and the precision.
4. Guard rails: a data quality monitor on the top features and a model quality monitor once churn labels land, plus a weekly review of precision@500. If precision@500 degrades while data quality is stable, retrain; if data quality alarms fire first, fix the pipeline.

Finally, run Clarify bias metrics (class imbalance, difference in positive proportions in predicted labels) across region and tenure segments before go-live, since a capacity-limited process makes whatever it skips invisible.

First establish what 999 means, because the treatment depends on the semantics. It almost always encodes "never logged in" or "unknown", not "999 days ago". Check its correlation with tenure and with churn - if never-logged-in users churn at a different rate, the sentinel carries real signal.

• Wrong: impute to the feature mean. That places never-logged-in users in the middle of the activity distribution, which is the opposite of the truth, and it makes the imputed rows indistinguishable from genuinely average users.
• Better: convert 999 to null, then add a binary has_never_logged_in indicator and let the tree split on the indicator. Set the null to a value outside the observed range (or let XGBoost's native missing handling route it), so the model can separate the group without a spurious 999-day ordinal claim.
• Why it matters even for trees: a split at days_since_last_login < 30 is unaffected, but every partial dependence plot, SHAP summary and any downstream linear or distance-based model is distorted by the fake magnitude. It also silently poisons any future feature that averages or bins this column.

Validate the choice on the time-based holdout: compare AUC/PR-AUC for sentinel-as-value versus sentinel-as-missing-plus-indicator, and check that the change in SHAP attribution is explainable to the business.