Why the Fraud Model Collapsed in Production
A payments company scores 1.2 million card transactions a month, of which 0.70% (about 8,400) are fraudulent. Features include amount, merchant category, device age and days since card issue. Offline, a random 5-fold stratified cross-validation over 18 months of history gives ROC-AUC 0.994 and average precision 0.91, and the model ships at the default 0.5 probability threshold. In production it raises about 900 alerts a day (27,000 a month), of which analysts confirm 6% as genuine fraud, and the review team is drowning. Two defects are found. First, the model's strongest feature is 'prior disputes on this card', computed by a nightly job that joins a table already containing the current month's disputes. Second, 3% of rows are exact duplicates produced by retried payment messages, and these duplicates straddle the train and validation folds. Each missed fraud costs about ₹4,500; each manual review costs ₹120.
1. Reconcile the offline average precision of 0.91 with the production precision of 0.06. Compute the implied recall and name the mechanisms.
Read the worked answer
Start from the production numbers. 27,000 alerts a month at 6% precision gives about 1,620 confirmed frauds. Against 8,400 actual frauds that is a recall of roughly 19%, not the near-perfect picture the offline metrics suggest. So the operating point is bad on both axes at once, which no single threshold move can explain.
The mechanisms are three distinct failures. (i) Target leakage: 'prior disputes on this card' is computed from a table that already includes the label window, so at training time the feature partially encodes the outcome. Offline the model appears to predict fraud from a variable that already knows the answer; at scoring time that variable is a month stale, so the signal vanishes. (ii) Duplicate rows in both train and validation make the validation set a near-copy of the training set, inflating apparent generalisation - 3% exact duplicates is small over all folds but concentrated enough in the positive class to matter. (iii) Random splitting ignores temporal drift: fraud patterns, merchant mix and thresholds move month to month, so a random split validates on the future relative to the training rows.
AUC 0.994 is also a poor guide under 0.7% prevalence: ROC-AUC is dominated by the large negative class and stays high even when precision at any usable threshold is low.
2. Design the evaluation protocol and feature pipeline that would have caught this before launch.
Read the worked answer
Split by time, never at random: train on months 1–12, tune on 13–15, and report on 16–18 as a held-out future block. Never let a row's feature state be computed from data after that row's timestamp - implement point-in-time joins, so 'prior disputes' is evaluated as of the transaction time. If the as-of feature is expensive, freeze a feature snapshot per scoring day and reuse it for both training and serving so the two are byte-identical.
Deduplicate on a message key before splitting, and verify no key appears in more than one block.
Choose metrics that respect prevalence: average precision (PR-AUC), precision at the review capacity (e.g. precision@900/day), and recall at that same capacity. ROC-AUC can be reported for continuity but not used for selection. Always report the base rate alongside.
Add a train/serve skew check: recompute the top 20 features from the serving path on a sample of historical days and diff them against the training values. Leakage shows up here as a large discrepancy. Finally, hold out a recent month and re-score it monthly to track drift; a model whose held-out precision decays faster than the drift monitor predicts is a model with a pipeline bug, not a modelling bug.
3. Derive the cost-optimal decision threshold and explain how the 900-per-day review capacity interacts with it.
Read the worked answer
Let p be the model's calibrated probability of fraud. Flagging costs ₹120 if the transaction is legitimate and saves ₹4,500 if it is fraudulent. Expected cost of flagging is (1−p)·120; expected cost of not flagging is p·4,500. Flag when the former is smaller:
(1−p)·120 < p·4500 ⟹ 120 < p·4620 ⟹ p > 120/4620 = 0.02597.
So the cost-optimal threshold is about 0.026, not 0.5. The 0.5 default is a classification convention with no economic meaning here, and it is why recall sits near 19%.
Note what 0.026 does and does not mean. At scores exactly at the threshold the odds are roughly 1 in 37, so marginal precision is low; average precision over the flagged set is higher because higher-scored cases are better. Do not confuse the threshold with the resulting precision.
Capacity is a separate, binding constraint. If the threshold of 0.026 flags more than 27,000 transactions a month, the review team cannot process them all, and the real operating point is 'top 27,000 by score'. Compute the score at rank 27,000; if it exceeds 0.026 the capacity binds and the threshold is capacity-determined, so the correct actions are to expand the review team or add an automatic block tier above some higher score, rather than to accept a threshold the business cannot staff.