Fraud model with 0.8% positives, a 500-alert budget, and a new payment provider
A payments team trains a fraud classifier on 12 million transactions from the past two years; 0.8% are fraud. The current model reports ROC AUC 0.94 and accuracy 99.2%, and it was deployed at a 0.5 threshold. Investigators say it 'catches almost nothing'. Only 40 confirmed fraud cases exist in the last 30 days, and the review team can handle 500 alerts per day. The team split the data randomly, but a new payment provider was added 90 days ago and behaves differently. Leadership wants to know whether to trust the offline numbers before the next release, and the model retrains weekly.
1. Which metrics should replace ROC AUC and accuracy, and why do the current numbers look healthy while the model underperforms?
Read the worked answer
Both current metrics are dominated by the 99.2% negative class and are threshold-free.
- Accuracy of 99.2% is barely better than predicting 'never fraud' (99.2%). It is uninformative here.
- ROC AUC of 0.94 measures ranking across all thresholds. It rewards the model for confidently ranking the easy, obvious negatives and says nothing about the operating point.
Replace them with:
- PR-AUC (average precision) as the headline ranking metric - it weights the positive class directly.
- Recall at a fixed alert budget. The team can review 500 alerts/day out of ~16,400 daily transactions, an alert rate of ~3%. So compute recall at 3% of the highest-scored transactions, or equivalently precision-recall at that cutoff. This is the number investigators actually experience.
- False positives per day at that budget, so the cost of the recall is visible.
A model can have ROC AUC 0.94 and still, at threshold 0.5, flag almost nothing - because 0.5 is an arbitrary cut on a calibrated logit where virtually no transaction ever reaches 0.5 probability at a 0.8% base rate. The threshold should be set from the alert budget, not defaulted.
2. How should the evaluation split be constructed, and what does the random split hide?
Read the worked answer
Replace the random split with a time-based split: train on everything up to a cutoff, hold out the most recent 90-180 days, and report metrics on that holdout only.
The random split hides two things.
- Temporal leakage and drift. Random splitting puts transactions from the same week in both train and test. The model is scored on the era it was trained on. Production retrains weekly precisely because the fraud landscape moves; a random split will never show that decay. Worse, if the same card, device, or merchant appears on both sides, the model memorises identities rather than behaviour.
- The new payment provider. Because it only exists in the last 90 days, a random split puts it in both train and test, so its distinct behaviour is learned and then flattered. With a time-based split, provider-new traffic sits entirely in holdout and you see the truth: likely a segment with much weaker recall.
Also group-split by card or customer if any entity recurs, and report metrics per provider and per segment rather than only in aggregate - an aggregate number can look acceptable while the new provider is near-useless.
3. With only 40 confirmed frauds in the last 30 days, how do you produce a defensible estimate of performance at the 500-alert budget?
Read the worked answer
First, question the denominator. Fraud labels mature slowly - chargebacks are typically confirmed 60-90 days after the transaction. The last 30 days are therefore under-labelled, and recall computed on them is biased downward (or noise, depending on how partial labels were treated). Either restrict evaluation to a fully matured window or apply a maturation correction per age bucket.
Second, widen the window. Take 6-12 months of matured data whose provider mix matches today's - that is the only way to get a sample where recall-at-budget is estimable. 40 positives gives a recall confidence interval roughly ±15 percentage points; that is far too wide to support a release decision.
Third, report uncertainty explicitly: bootstrap confidence intervals on recall at the 3% alert rate, stratified by provider and transaction channel, and always paired with the false-positive count so the trade-off is visible.
Finally, define the decision rule before looking: for example, ship only if the lower bound of recall-at-budget exceeds the incumbent's point estimate on the same matured holdout. Do not ship on an aggregate that hides the new provider's segment.