Evaluating a risk-scoring model against fixed scanner capacity
A container port screens 25,000 containers a month and can physically scan 1,000 of them. Under the current targeting rules, 4% of containers are scanned (1,000 per month) and 180 of those yield a detection, meaning a seizure or a revenue assessment above the threshold. An intelligence-led model was trialled on 5,000 containers, all of which were examined so that ground truth was known. The model flagged 300 containers; 120 of those were true detections. Of the 4,700 not flagged, 30 were detections later found by other means. Scanning costs 90 per container. Average revenue recovered per true detection is 6,500. Scanning capacity is fixed and will not increase.
1. Calculate the model's precision, recall and flag rate, and explain what each figure means for the port.
Read the worked answer
Confusion matrix from the trial: true positives 120; false positives 300 - 120 = 180; false negatives 30; true negatives 4,700 - 30 = 4,670. Total actual detections = 120 + 30 = 150.
- Precision = TP / flagged = 120 / 300 = 40%. Four in ten scanned containers will justify the scan.
- Recall = TP / actual detections = 120 / 150 = 80%. The model catches four-fifths of what is there; one-fifth is missed.
- Flag rate = 300 / 5,000 = 6% of the screened population.
- Prevalence in the trial = 150 / 5,000 = 3%.
Interpretation. Precision is the number that matters financially, because scanning capacity is fixed: every false positive consumes a slot that could have gone to a real detection. Recall matters for enforcement reputation and for deterrence, but a high recall is useless if precision is so low that the 1,000 slots are wasted.
Note the base rate. With 3% prevalence, a 6% flag rate and 80% recall are consistent: 0.8 x 3% = 2.4% of all containers are true positives, and 2.4 / 6 = 40% precision.
2. Compare the current rules with the model on detections, cost per detection and revenue, holding capacity fixed at 1,000 scans.
Read the worked answer
Current rules:
- Scans: 1,000; cost 1,000 x 90 = 90,000
- Detections: 180
- Cost per detection: 90,000 / 180 = 500
- Revenue: 180 x 6,500 = 1,170,000
- Net contribution: 1,170,000 - 90,000 = 1,080,000
Model (assuming trial prevalence holds):
- Scans: 1,000; cost 90,000
- Precision 40%, so detections = 1,000 x 0.40 = 400
- Cost per detection: 90,000 / 400 = 225
- Revenue: 400 x 6,500 = 2,600,000
- Net contribution: 2,600,000 - 90,000 = 2,510,000
Comparison: detections up from 180 to 400; cost per detection down 55%; net contribution up by 1,430,000 per month.
The comparison is only valid because the scan budget is fixed. If capacity could expand, the marginal value of extra scans would fall as precision declines, and the optimum would move. With fixed capacity, the model is preferred on every measure.
3. What assumptions could make this comparison wrong, and what monitoring would you require before adopting the model?
Read the worked answer
Assumption 1: the trial population is representative. If the trial was run on a high-risk lane, 3% prevalence will overstate reality. Holding the 6% flag rate and 80% recall, precision at 1% prevalence would be 0.008 / 0.06 = 13%, giving only 133 detections per 1,000 scans - worse than the current 180. The single most important check is the prevalence of the population the model will actually score.
Assumption 2: the trial was blind. If officers knew the model's flags and searched flagged containers harder, the measured precision is inflated and the 30 missed detections understate the true miss rate.
Assumption 3: stability. Criminal adaptation, new routing or a change in commodity mix will degrade a model trained on last year's data. Detection performance decays without retraining.
Assumption 4: revenue per detection is constant. A model shifts the mix of what is found; 6,500 is an average that may not hold for a different detection profile.
Monitoring to require:
- A shadow-mode run for at least one quarter, flagging without acting, to measure precision on the live population.
- Monthly tracking of precision, recall where measurable, flag rate and revenue per scan.
- A random control sample of unflagged containers so that missed detections remain visible; without it, the model becomes unfalsifiable.
- A documented override and review process so officers can escalate outside the model, with overrides analysed for model error.
- A pre-agreed retraining trigger, for example precision falling below the current system's implied rate.