Wafer-defect classifier with high validation PR-AUC but poor line recall
A semiconductor fab pilots a wafer-map defect classifier. The training set contains 120,000 die images, of which 0.9% are defective. A random stratified split by image yields PR-AUC 0.91 and ROC-AUC 0.98. At threshold 0.5, validation recall is 0.88. In production, at the same threshold, recall falls to 0.42 over two weeks even though the defect rate is similar. Each wafer contributes 40 to 80 die images, and lots are processed in campaigns with different tools and lighting. The team retrained with more data and got the same validation score. They now plan to lower the threshold to 0.2 to recover recall. The line can review only 300 alarms per shift; current volume is 90 alarms per shift.
1. Explain how the validation result and the production recall can diverge so sharply. Identify the methodological flaws and propose a corrected evaluation protocol.
Read the worked answer
The random split by image leaks information: die images from the same wafer, and often the same lot and campaign, appear in both training and validation. The model can learn wafer-specific signatures, tool offsets, or lighting patterns rather than generalizable defect features, inflating PR-AUC and recall. Production introduces new wafers, lots, tools, and campaigns, so that memorization does not transfer. The high ROC-AUC is also less informative under 0.9% prevalence than PR-AUC.
Corrected protocol: split by wafer and lot, not by image, using grouped cross-validation; add a time-based holdout that simulates deployment on future campaigns. Report PR-AUC, recall at a fixed false-alarm budget, calibration error, and performance by tool and lighting condition. Select the threshold inside each training fold and evaluate on the held-out group, never on the same random split. If performance collapses on the grouped split, the original 0.91 was not a deployment estimate. Retraining with more randomly split data will not fix this; the split and the sampling unit must change.
2. Evaluate the proposal to lower the threshold to 0.2. What operating policy and monitoring would you recommend instead?
Read the worked answer
Lowering the threshold is not a substitute for a valid model. At threshold 0.5 the line sees 90 alarms per shift; dropping to 0.2 will increase alarms and may exceed the 300-per-shift review capacity, while precision will fall. The correct policy is to choose the threshold from a grouped, time-based holdout so that expected alarms per shift meet the 300 budget at the best achievable recall, then calibrate scores on recent production data. If 0.2 is the only way to reach acceptable recall, the model probably needs retraining on grouped data or domain adaptation, not just a lower cut point.
Monitor score distribution, alarm volume, defect rate, and recall from reviewed labels by lot and tool. Set drift alerts when the score distribution or alarm rate shifts beyond the validation range. Re-estimate calibration and threshold on a rolling schedule, and keep a human review loop for borderline scores. Report recall at the fixed alarm budget rather than an unconstrained threshold, because that is the decision the line can actually execute.