Calibrating a Rare-Event Risk Model Under a Fixed Precision Budget
A fraud team deploys a gradient-boosted classifier on 1.2M monthly transactions with a 0.4% positive rate. The model outputs a probability, but the downstream rules engine only accepts a hard flag. Operations insists on precision of at least 0.60 because each false positive triggers a manual review costing 90 seconds of an analyst's time. The team has 2,000 analyst-hours per month. On last month's holdout, at the threshold that maximizes F1, the model flagged 14,000 transactions with precision 0.41 and recall 0.72. A reliability plot shows the top decile of scores is well-calibrated but the 0.5-0.8 band is systematically overconfident. Leadership asks whether the model is 'good enough' and what to change before launch.
1. Using the holdout numbers, determine whether the current F1-optimal operating point fits the analyst-hour budget, and state how many true positives are captured there.
Read the worked answer
The positive class has 0.4% of 1.2M = 4,800 actual frauds. At recall 0.72, true positives = 0.72 x 4,800 = 3,456. Precision 0.41 means flagged count = TP / precision = 3,456 / 0.41 = 8,429, so false positives = 8,429 - 3,456 = 4,973. Each review costs 90 s = 0.025 h, so 8,429 reviews need ~211 analyst-hours.
That is well within 2,000 hours. The binding constraint is not analyst capacity but the precision target: 0.41 < 0.60. So the operating point is affordable but non-compliant. The team is not budget-limited; it is precision-limited, and should move the threshold up rather than add capacity.
2. The team considers raising the threshold to hit precision 0.60. Explain the trade-off they should quantify, and why the miscalibrated 0.5-0.8 band complicates threshold selection.
Read the worked answer
Raising the threshold trades recall for precision. At precision 0.60 the flagged set shrinks; if precision rises from 0.41 to 0.60 while the score distribution is roughly fixed, flagged count falls and recall drops materially. The team should report the resulting recall and the absolute number of missed frauds, not just the ratio, since 0.4% base rate means small recall losses are large in count.
The miscalibration matters because threshold selection usually assumes score = probability. If the 0.5-0.8 band is overconfident, a nominal 0.6 cutoff does not actually deliver 0.60 precision; the realized precision will be lower than the score suggests. They should either recalibrate (isotonic or Platt on a held-out set) before choosing the threshold, or select the threshold empirically on validation data by scanning realized precision, then confirm on a fresh holdout. Calibration should be fit on data disjoint from threshold selection to avoid optimistic bias.
3. Leadership proposes 'just lower the review cost by auto-clearing low-value transactions.' Evaluate this as a fix for the precision problem.
Read the worked answer
Auto-clearing low-value transactions reduces the cost of a false positive but does not change precision - the fraction of flags that are wrong. If the 0.60 precision requirement is a policy/compliance rule, cost reduction does not satisfy it. If the requirement is really an economic one (review cost vs. fraud loss), then reframing is legitimate: the correct decision rule maximizes expected value, flagging when P(fraud) x loss > review cost. Under that framing, cheaper reviews justify a lower threshold, not a higher one, which worsens precision while possibly improving net benefit.
So the team must clarify which constraint is real. If it is economic, replace the precision target with an expected-cost objective and recompute the optimal threshold. If it is a hard policy floor, cost engineering is irrelevant and the threshold must move. Conflating the two is the error to catch.