Reporting a Late-Payment Model Without the Accuracy Trap
A finance shared-services group built a model to flag invoices likely to be paid more than 60 days late. On a held-out test set of 50,000 invoices, 2,000 were actually late (4%). At the chosen threshold the model flags 3,000 invoices: 900 of them truly late and 2,100 false alarms; it misses 1,100 late invoices. The sponsor wants to put 94% accurate in the board deck and also wants the threshold set so we catch every late invoice. Collections can work roughly 3,000 flagged invoices per month, and each late invoice costs about USD 900 in financing and follow-up.
1. The sponsor's 94% figure is misleading. Show why with the base rate, and compute what the accuracy actually is.
Read the worked answer
The arithmetic
- Correct predictions: 900 true late + 45,900 true not-late = 46,800 of 50,000 → 93.6% accuracy. So 94% is also slightly overstated, but that is the smaller problem.
Why the number is meaningless here
- Only 4% of invoices are late. A model that predicts nothing is ever late, every time, scores 96% accuracy - better than this model - and is completely useless.
- Accuracy is dominated by the majority class. With a 4% event rate, any headline accuracy above 95% is nearly guaranteed and tells the board nothing about whether the flag is doing work.
The honest framing Of the 3,000 invoices the model flags, 900 are genuinely late and 2,100 are not - a 70% false-alarm rate. That is the number the collections team will experience, and it is the number the board should see alongside the benefit.
Also correct the sponsor's number before it reaches a board pack; a stated figure that does not reconcile with the underlying test set destroys credibility the first time someone checks it.
2. Compute precision and recall at the current threshold, then explain what necessarily happens to precision if the threshold is moved to catch every late invoice.
Read the worked answer
At the current threshold
- Precision = 900 / 3,000 = 30%. Seven of ten flags are false alarms.
- Recall = 900 / 2,000 = 45%. More than half of late invoices are missed.
- Specificity = 45,900 / 48,000 = 95.6%.
Catching every late invoice (recall = 100%) To flag all 2,000 late invoices, the threshold must be lowered until every late case scores above it. Because late and not-late invoices overlap heavily in score, that threshold will also flag a large number of on-time invoices. Precision must fall - necessarily, not incidentally - because the denominator of precision grows faster than the numerator once you push past the easy true positives.
Magnitude Even a modest expansion shows the problem: to move recall from 45% toward 100%, flagged volume will exceed the 3,000-per-month capacity by a wide margin, so the operational constraint makes 'catch everything' impossible anyway.
Reframe the goal 'Catch every late invoice' is not a threshold setting; it is a statement about the relative cost of misses and false alarms. Make that trade-off explicit: a missed late invoice costs about USD 900, while a false alarm costs an analyst's review time. Compare the two and pick the threshold that minimizes total cost, not one that satisfies a slogan.
3. Design the reporting package for the board. What metrics, what framing, and how do you tie the threshold to the 3,000-invoice capacity?
Read the worked answer
Metrics to report (not accuracy)
- Precision at the operating threshold (30%), with the implied review workload: 3,000 flags → 2,100 false alarms.
- Recall (45%) and the dollar exposure of misses: 1,100 missed × USD 900 ≈ USD 990,000 in financing and follow-up cost.
- Lift over random: 30% precision versus a 4% base rate is a 7.5× concentration of true late payers in the flagged list. This is the clearest statement of value.
- Expected value at threshold: 900 caught × USD 900 avoided ≈ USD 810,000 in avoided cost, against the review capacity consumed.
- A curve, not a point: precision–recall across thresholds so the board can see the trade-off rather than a single flattering number.
Tie to capacity
- Rank all invoices by score and cut at 3,000, since collections can only work 3,000. Report the score at the cut and how flat the curve is near it - if the 3,000th and 4,000th invoices score almost the same, the capacity limit is close to binding and a small capacity increase buys little.
- Alternatively, set the threshold where marginal avoided cost equals marginal review cost, and then show what capacity that threshold requires. Present both and let the board choose capacity versus cost.
Framing discipline Lead with the decision (how many to review, at what precision, saving what), state the model's known weakness (misses 55% of late invoices), and put the confusion matrix in an appendix so the numbers are auditable.