Fraud Model Threshold, Leakage, and Metric Reporting
A retail bank deploys a gradient-boosted fraud detector trained on 2.1 million card transactions spanning 18 months, with a 0.9% fraud prevalence. Offline ROC-AUC is 0.97. At the live threshold of 0.50 the model produces 1,900 alerts per day with precision 0.71 and recall 0.18. Fraud operations wants recall of at least 0.60 while limiting alerts to no more than four times today's volume. A data scientist proposes lowering the threshold to 0.12, where offline evaluation shows recall 0.64 and precision 0.09. Separately, an auditor finds that the feature days_since_last_chargeback was computed using the chargeback date, which is recorded up to 60 days after the transaction. The model is retrained monthly on a rolling 18-month window, and the same pipeline feeds both training and backtesting. Fraud losses average $310 per confirmed fraudulent transaction, and each alert costs about $4 to investigate.
1. Does the proposed threshold of 0.12 satisfy the fraud team's constraints? Show the arithmetic and state what you would recommend.
Read the worked answer
At the current operating point, 1,900 alerts at precision 0.71 imply about 1,349 true frauds caught per day. With recall 0.18, total fraud attempts are roughly 1,349 / 0.18 ≈ 7,494 per day. A recall target of 0.60 means catching about 4,496 frauds per day. At precision 0.09, that requires 4,496 / 0.09 ≈ 49,960 alerts per day, roughly 26 times current volume - far beyond the four-times cap of 7,600 alerts.
The constraint can also be inverted: at 7,600 alerts and 4,496 true positives, precision would need to be about 0.59. The proposed operating point delivers 0.09. The proposal therefore fails on the alert-volume constraint by a wide margin, and the offline precision estimate may not even hold in production.
I would reject the threshold change as specified and instead pursue precision at the required recall: add features available at authorization time, use segment-specific thresholds by merchant category or channel, apply cost-sensitive learning with the $310 loss and $4 investigation cost, and route a mid-score band to lighter-weight review. Any threshold change should be evaluated against the alert budget, not recall alone.
2. Explain the effect of the days_since_last_chargeback feature on offline evaluation and production, and what must change before the next retrain.
Read the worked answer
This is target leakage through future information. A chargeback is recorded up to 60 days after the transaction, so at scoring time for a new authorization the feature value is unavailable or materially different from its training value. The model can learn that a short gap since a chargeback predicts fraud, but that is only knowable after the fact, so offline metrics are inflated and production performance will degrade.
Before retraining, I would enforce point-in-time correctness: compute every feature with an as-of join keyed to the transaction timestamp, using only data available at authorization. Exclude or redesign this feature, and audit the remaining feature set for the same defect. Backtesting must respect label maturity - exclude the most recent 60 days from training labels, or use a time-based split with a gap. I would also compare offline and production precision-recall on the same period to quantify the inflation, and add a feature-availability test to the pipeline so a future change cannot silently reintroduce post-event fields.
3. Which headline metric should be reported to the fraud steering committee, and how does the 0.9% prevalence change the interpretation of ROC-AUC?
Read the worked answer
Report precision-recall metrics at the operating point, not ROC-AUC alone. Specifically: PR-AUC, recall at the fixed alert budget, precision at that budget, expected fraud loss avoided versus investigation cost, and a calibration plot.
With 0.9% prevalence, ROC-AUC is dominated by the large number of true negatives. A model can score 0.97 AUC while producing poor precision in the alert band the business actually uses, because a small false-positive rate applied to 99.1% of traffic still generates many alerts. Precision-recall curves are sensitive to the positive class and reflect the trade-off the fraud team faces.
I would present a table of operating points: alerts per day, recall, precision, fraud dollars caught, and investigation cost, with the four-times cap marked. That makes the decision about acceptable analyst workload explicit and prevents a headline AUC from masking an unusable threshold.