Fraud Model Validation with Repeated Cards and Bursty Fraud
A payments processor has 18 months of card-not-present transactions: 42 million rows and a 0.28% fraud rate. The data science team trains a gradient-boosted classifier on a random 80/20 split and reports ROC-AUC 0.986 and PR-AUC 0.71. When they set the threshold to flag 1% of transactions, precision is 0.19 and recall is 0.68. Operations currently reviews 12,000 alerts per month; 85% are false positives, and the review team cannot absorb more than 15,000. The same card appears in both training and test sets many times, and fraud rings often test cards in short bursts. Features include amount, merchant category, hour, and card transaction count in the last 24 hours. The business wants fewer false positives without missing more fraud, and the model owner wants to know whether the offline estimate can be trusted before a pilot.
1. The team wants to know whether the 0.986 ROC-AUC can support a pilot decision. Explain why that number may be optimistic and interpret the ROC-AUC versus PR-AUC gap.
Read the worked answer
Leakage from the split. A random row-level split puts the same card in train and test. The model can memorize card-specific behavior, and because fraud rings test cards in bursts, near-duplicate transactions land on both sides. The reported AUC partly measures card recognition, not fraud generalization.
Class imbalance affects the metrics differently. ROC-AUC is computed over all thresholds and is dominated by the large negative class; a model can rank many good negatives and still look strong while doing poorly on the rare positive class. PR-AUC ignores true negatives, so it is more sensitive to false positives among the alerts the business actually reviews. A large ROC-AUC with PR-AUC 0.71 is a warning that the operating range is weak.
What to conclude. The 0.986 cannot support a pilot by itself. Re-estimate on an out-of-time, card-grouped split. If PR-AUC falls materially, the current number is leakage-inflated. The decision should use precision and recall at the review capacity, not a threshold-free AUC.
2. Recommend a validation design and a primary metric set for deciding whether to pilot.
Read the worked answer
Use a time-based split with a gap: train on months 1 through 12, validate on months 14 and 15, test on months 16 through 18, leaving month 13 as a purge gap so 24-hour and burst features do not cross the boundary. Group by card hash so no card appears in more than one fold; use GroupKFold for hyperparameter tuning inside the training window.
Report PR-AUC, precision at the 15,000-alert budget, recall at a fixed precision such as 0.50, and calibration error. Bootstrap by card and by time block to get confidence intervals, because transaction-level intervals will be too narrow. Also report performance separately for new cards, high-velocity cards, and each merchant category.
The pilot decision should require the lower bound of precision at 15,000 alerts to exceed the current 0.15 and recall to be no worse than 0.68. A single aggregate ROC-AUC should be secondary because it does not reflect the review queue.
3. How should the team choose the operating threshold given the 15,000 alert cap and review cost?
Read the worked answer
Start from the operational constraint: the review team can handle 15,000 alerts per month, so rank transactions by score and set the cutoff at the 15,000th highest score in a recent validation window, then recalibrate monthly. Do not assume 0.5 or any fixed probability is meaningful. Build a cost matrix with the cost of a review, expected fraud loss avoided, and customer friction from a false decline. If review cost is low relative to fraud loss, the optimal point may be above 15,000 alerts, which means the constraint is a business choice to revisit.
Evaluate precision at k and recall at k, not only threshold-free metrics. Because fraud prevalence shifts, monitor alert volume and precision weekly; if volume exceeds capacity, tighten the cutoff and route overflow to a secondary queue rather than silently dropping alerts. Keep the score calibrated so the threshold remains interpretable across retraining cycles.