Cost-Sensitive Threshold Selection Under Delayed Labels
A utility deploys a gradient-boosted classifier to rank 200,000 distribution transformers for failure within 30 days. Only 1.2% fail in the window. On a held-out year, the model has ROC-AUC 0.90. At the default 0.5 threshold it catches 1,800 of the 2,400 failures and raises 18,000 false alarms. A missed failure costs about $1,200 in emergency repair and outage penalties; a false alarm costs $40 for inspection. The maintenance team can inspect at most 25,000 transformers per month and wants high recall. The model was trained on 2019-2022 data; in 2024 a heat wave and new EV charging load changed the failure pattern. Monitoring shows the score distribution shifted upward, but labels arrive only after 60 days.
1. Compute precision, recall and expected cost at the 0.5 threshold. Compare a lower threshold with 92% recall and 60,000 false alarms, and a higher threshold with 45% recall and 3,000 false alarms. Which threshold is best under the inspection cap?
Read the worked answer
At 0.5, TP = 1,800, FN = 2,400 - 1,800 = 600, FP = 18,000. Precision is 1,800/(1,800 + 18,000) = 9.1%, and recall is 1,800/2,400 = 75%. Cost is 600 x $1,200 + 18,000 x $40 = $720,000 + $720,000 = $1.44 million, with 19,800 inspections, under the 25,000 cap. At the lower threshold, TP = 0.92 x 2,400 = 2,208 and FN = 192. Cost is 192 x $1,200 + 60,000 x $40 = $230,400 + $2.4 million = $2.63 million, and 62,208 inspections exceed capacity. At the higher threshold, TP = 0.45 x 2,400 = 1,080 and FN = 1,320. Cost is 1,320 x $1,200 + 3,000 x $40 = $1,584,000 + $120,000 = $1.704 million. The 0.5 threshold has the lowest cost of the three and fits the inspection cap, so lowering the threshold to chase recall is not justified because false-alarm cost and capacity dominate.
2. The score distribution has shifted upward in 2024. Explain why this is not evidence of improved detection and propose a monitoring and retraining plan that respects the 60-day label delay.
Read the worked answer
An upward score shift means the model is raising more alarms, not necessarily finding more true failures. Without labels, precision cannot be computed, and the shift may reflect a changed feature distribution from heat and EV load rather than a change in failure risk. Monitor input features such as temperature, loading and charging profiles, plus calibration and alarm volume. Track inspection yield, the fraction of inspected assets that actually fail, as a leading indicator. Backfill labels after 60 days and compare precision at the operating threshold with the held-out baseline. Use shadow deployment of a retrained model on recent data and recalibrate the threshold on a rolling validation window. Trigger review if inspection yield falls materially or if alarm volume exceeds the 25,000 cap.
3. The team insists on 90% recall while staying within 25,000 inspections. Show whether this is feasible with the current model and state what must change if it is not.
Read the worked answer
At 90% recall, TP = 2,160 and FN = 240. The inspection cap leaves at most 25,000 - 2,160 = 22,840 false alarms, so required precision is 2,160/25,000 = 8.6%. At the 0.5 threshold the model has 9.1% precision at 75% recall, but to move recall from 75% to 90% the threshold must drop and precision will fall below 8.6%, breaching the capacity cap. The target is therefore not feasible by threshold tuning alone. The utility must improve the model, for example with recent labelled data, better features for EV loading and thermal stress, or a two-stage screening model. Alternatively, it must raise the inspection capacity or accept a lower recall. Threshold selection cannot create separability that the model does not have.