Fraud Model with a 99.4% Accuracy Claim
A payments team trains a fraud classifier on 2.4 million card transactions spanning 18 months, of which 14,400 (0.6%) are confirmed fraud. Features include merchant category, amount, hour, device age, and chargeback_flag, which the disputes team populates about 30 days after the transaction. The team uses a random 80/20 split, trains gradient-boosted trees, and reports 99.4% accuracy and 0.99 ROC-AUC on the holdout. At the proposed 0.5 threshold the model flags 9% of all transactions. Average fraud loss is USD 180 per case and a manual review costs USD 6. The risk committee asks for a pre-deployment review; the data scientist argues the metrics prove the model works and wants to ship it this week.
1. Explain why 99.4% accuracy and 0.99 ROC-AUC are not sufficient evidence for deployment.
Read the worked answer
Compute the trivial baseline first. Predicting 'not fraud' for every transaction gives 14,400/2,400,000 = 0.6% positives, so an always-negative classifier scores 99.4% accuracy. The reported accuracy is exactly the baseline and therefore says nothing about fraud detection.
ROC-AUC is also weak evidence under extreme imbalance. With a very large negative class, false positives accumulate slowly across thresholds, so AUC can stay near 0.99 while precision at the chosen operating point is poor.
Use the flag rate to check precision. At 9%, the model raises 216,000 alerts. If it catches 90% of fraud, that is 12,960 true positives, so precision is 12,960/216,000 ≈ 6%: about 19 of every 20 alerts are false. The cost picture is 216,000 reviews × USD 6 = USD 1.30 million, against 12,960 × USD 180 = USD 2.33 million in avoided fraud losses. The net is positive but operationally implausible, and the 0.5 threshold was never justified. Report precision-recall AUC, recall at the review team's actual capacity, and expected cost per threshold.
2. Identify the data leakage and evaluation-design flaws, and describe the corrected split.
Read the worked answer
chargeback_flag is recorded roughly 30 days after the transaction and is essentially a proxy for the label. At scoring time it is unavailable, and during training it lets the model see the outcome. It must be removed, along with any other feature populated after the decision point.
The random 80/20 split has two problems. First, the same cardholder or merchant can appear in both train and test, so the model can memorize entity-specific patterns and inflate the score. Second, fraud patterns drift, and a random split lets future transactions inform past ones.
Use a time-based split: train on the first 14 months, validate on the next two, and test on the final two, with a gap of at least the chargeback window between train and test to avoid label contamination. Group by card number when splitting. Audit every feature for availability at authorization time and rebuild the pipeline. Expect measured performance to drop; that drop is the honest estimate.
3. Design the deployment gate and monitoring plan for the first eight weeks.
Read the worked answer
Start in shadow mode: score live transactions, log decisions, and compare against existing rules without blocking anything. Set the threshold from review capacity, not from 0.5. If the team can handle 2,000 reviews per day, choose the threshold where expected alerts fit that capacity and precision is highest.
Gate criteria should be explicit: precision at capacity above 25%, recall above 50%, alert volume within capacity, and no reliance on post-outcome features. Validate on the time-based holdout, not the random split.
Monitoring should track score distribution, feature drift, label delay, and weekly precision and recall once chargebacks mature. Add a manual review sample of low-scoring transactions to estimate false negatives. Retrain on a fixed cadence and after any drift alert. Do not ship this week; the leakage alone invalidates the reported metrics.