Imbalanced Fraud Labels and a Precision Target
A payments team built a fraud detector on 2 million transactions, of which 0.4% are fraudulent. The model achieves 92% recall at a threshold that flags 3% of all transactions. The operations team can manually review only 0.8% of daily volume. Each false positive costs about 15 minutes of analyst time; each missed fraud costs on average 400 in losses plus chargeback fees. The current rule-based system flags 1.5% of volume with 55% recall. A stakeholder proposes optimizing for accuracy, noting the model is already above 99% accurate. Another proposes raising the threshold until flagged volume hits 0.8%. The team has a held-out test set and a cost matrix but has not yet computed expected costs.
1. Explain why accuracy is a misleading objective here and what metric should drive model selection.
Read the worked answer
With 0.4% positives, a model that predicts 'never fraud' scores 99.6% accuracy while catching nothing. Accuracy is dominated by the majority class, so it cannot distinguish a useful detector from a useless one. The stakeholder's 'above 99% accurate' observation is therefore close to meaningless. Model selection should use metrics that reflect the operating constraint and the cost asymmetry: precision-recall AUC for ranking quality, and expected cost or expected value at the chosen threshold. Because only 0.8% of volume can be reviewed, precision at that review capacity matters most: of the flagged cases, how many are truly fraud, and how much fraud is caught. Recall alone (92%) hides that flagging 3% of volume is nearly four times the review capacity, so most flags would go unreviewed and the queue would back up.
2. Using the numbers given, compare the model at the 0.8% flag rate against the rule-based system, and state what additional calculation is needed before deciding.
Read the worked answer
At 0.8% of 2 million, capacity is 16,000 reviews. The rule-based system flags 1.5% (30,000) with 55% recall. Total fraud is 8,000; the rules catch 4,400. The model at 3% flags 60,000, far beyond capacity, so it must be thresholded down. At 0.8% flags (16,000), recall will fall below 92%, but the setup does not give the precision-recall curve, so exact recall at that point is unknown and must be read from the test set. What we can say: the model's ranking is likely better, so at equal review volume it should catch more fraud than the rules, but this must be verified. The missing calculation is expected cost at each threshold: false positives times analyst cost plus false negatives times 400-plus-fees, compared across the model and the rules, including the cost of unreviewed flags if volume exceeds capacity.
3. The stakeholder insists on matching the 0.8% review capacity exactly. What risks does that create, and how would you mitigate them?
Read the worked answer
Pinning the threshold to a fixed flag rate makes the operating point drift with traffic mix: on a busy day the same percentage means more absolute flags than analysts can handle, and on a quiet day capacity sits idle. It also ignores that the cost-optimal threshold depends on the score distribution, not on a volume quota. Mitigation: set the threshold from expected cost on validation data, then use a capacity-aware queue that ranks by score and reviews the top N, reporting both the fraction reviewed and the estimated fraud missed below the cutoff. Monitor precision at the cutoff weekly; if precision falls, the threshold is stale. Keep a fast lane for very high scores and a sampled audit of unflagged transactions to estimate residual fraud, so recall is not invisible. Document that the 0.8% figure is a capacity constraint, not an objective.