Recall Collapse After a Feature-Store Swap
A card issuer's fraud team trains a gradient-boosted tree on 18 months of transactions, roughly 12 million per day with 0.42% fraud. Offline AUC is 0.93 and has been stable for a year. In January the team replaced a hand-built merchant-category feature with an upstream feature-store table. Offline recall at the fixed alert budget of 0.5% of transactions fell from 61% to 43%, while AUC moved only from 0.932 to 0.929. The upstream table has a 24-hour backfill lag, and its category mapping changed for 8% of merchants in December. The team's validation split is a random 80/20 over all 18 months, the model is retrained monthly with unchanged hyperparameters, and the score threshold is recomputed from the training set each time. Business wants alert volume held constant; fraud ops reviews every alert manually and cannot absorb more than the current 0.5%.
1. Which offline metric should govern model selection here, and why is AUC misleading in this situation?
Read the worked answer
The alert budget is fixed at 0.5% of transactions, so the operating point is fixed. That makes this a top-k ranking problem, and the governing metric should be [email protected]%, with [email protected]% reported alongside - they are two views of the same cut, and fraud ops feels both.
AUC is misleading for two distinct reasons. First, it is computed over the entire ranking, so it is dominated by easy negatives far from the threshold. A model can degrade badly in the top 0.5% and lose almost nothing in AUC. Second, it is threshold-free, so it says nothing about behaviour at the volume the team can actually review.
The observed pattern - AUC 0.932 to 0.929 while [email protected]% falls from 61% to 43% - is the classic signature of a score distribution that shifted near the cut, or a top-of-ranking that got worse while the bulk stayed fine. Reporting AUC alone would have called this a non-event.
Add PR-AUC as a secondary summary for stability across cuts, and a cost-weighted metric if fraud loss per case and review cost per alert are known, because recall@k treats all alerts as equally valuable. Do not tune the threshold down to recover recall: the 0.5% budget is a staffing constraint, not a free parameter, and silently raising it just moves the failure into the review queue.
2. Diagnose the January regression and redesign the validation split that produced the misleadingly stable numbers.
Read the worked answer
There are two separate defects here, one causal and one methodological.
Diagnosis. The timing is the tell: the regression arrived with the feature-store swap, and the upstream table has a 24-hour backfill lag. If training joins are not point-in-time correct, the most recent day of every training window carries stale or default category values, so the model learns a mapping that does not exist at serving time. Add the December mapping change for 8% of merchants: for those rows the relationship between category and fraud is genuinely different, and a model trained on the old mapping is systematically wrong for them. Confirm by slicing [email protected]% by merchant-category-changed and by row age. If the loss concentrates in the changed 8% or the last day of the window, the cause is established. Compare feature distributions from the backfill path against the online path for the same merchant IDs - a direct train/serve skew check.
Validation split. A random 80/20 over 18 months leaks future information into training and hides temporal drift entirely, so the offline number was optimistic even before January. Replace it with a time-ordered split and a gap: train on months 1–14, validate on 15–17, with an embargo at least as long as label maturity plus the 24-hour backfill lag. Then score champion and challenger on the same holdout at the fixed alert budget, never at AUC.
3. The model is retrained monthly with frozen hyperparameters and a threshold re-derived from the training set. Propose a tuning, retraining and calibration policy.
Read the worked answer
Monthly retraining with frozen hyperparameters is a reasonable default but it is not a policy: it neither reacts to drift nor bounds the cost of tuning, and re-deriving the threshold from training data each month silently changes the effective alert volume.
Retraining trigger. Keep monthly as the floor and add event-driven retrains on data-quality and drift signals: upstream schema or mapping changes (as in January), a sustained gap between realised [email protected]% and the champion's backtest, or a large shift in transaction mix. Gate it - the pipeline should refuse to promote if feature null rates, category coverage or point-in-time join checks fail. A retrain on corrupted features propagates the corruption faster.
Tuning cadence. Re-tune hyperparameters quarterly, not monthly; the search is expensive and the optimum moves slowly. Use rolling-origin time-series CV with [email protected]% subject to a precision floor as the objective, and log every trial as a child run so the choice is auditable.
Calibration. Keep a calibration step (isotonic or Platt on the most recent mature window) so the score-to-threshold mapping survives retrains. Without it, each retrain changes the alert volume, and fraud ops will notice that before the model owner does.
Promotion. Champion/challenger on a shared evaluation window, with automatic rollback if live precision at fixed volume drops beyond an agreed margin.