A Defect Classifier with 97.8% Accuracy and 0.14 Recall
A factory has 48,000 images from an optical inspection station; 2.1% show a surface defect. A ResNet-50 binary classifier trained with cross-entropy reaches 97.8% validation accuracy, AUC 0.81, and recall 0.14 at the default 0.5 threshold. The validation set was created by randomly splitting images, but each physical board contributes 6–10 images taken milliseconds apart, and boards from the same production lot were split across train and validation. The line manager wants to add dropout and train 50 more epochs because the model is overfitting. A new production line with different lighting will be added next month. The business cost of a missed defect is about 40 times the cost of a false alarm. Inference must stay under 20 ms on the existing CPU.
1. Is the model overfitting? Assess the evidence and the manager's proposal.
Read the worked answer
The evidence does not support overfitting as the primary diagnosis.
The majority-class rate is 97.9%, and validation accuracy is 97.8%. A model that predicts negative for every image would score almost identically. Accuracy is therefore uninformative here. Recall of 0.14 at threshold 0.5 means the model misses 86% of defects, while AUC 0.81 shows it retains some ranking signal. That pattern is consistent with severe class imbalance and a badly placed threshold, not with high variance.
Adding dropout and 50 more epochs treats a symptom that has not been demonstrated. Dropout may reduce capacity further on an already under-detected minority class, and more epochs will not change the ranking quality much if the loss is dominated by negatives.
The correct next step is to inspect the confusion matrix and precision-recall curve, not the accuracy. If precision at high recall is non-trivial, threshold tuning alone may deliver most of the business value at zero inference cost. Overfitting should be tested by comparing train and group-held-out validation curves, not assumed from an accuracy gap that does not exist.
2. Diagnose the validation protocol and propose a corrected evaluation.
Read the worked answer
The random split leaks information in two ways. First, 6–10 near-identical frames of the same board land on both sides, so the model can recognize the board rather than the defect. Second, boards from the same lot share surface finish, lighting, and process conditions, so lot-level identity leaks too. AUC 0.81 is therefore optimistic and cannot be trusted as a production estimate.
Corrected protocol:
- Group split by board so all frames of a board are in exactly one split.
- Group split by lot for the final estimate, or at minimum hold out entire lots, because the new line is a distribution shift.
- Time-based split if the new line data arrives later; this simulates deployment better than any random split.
- Report precision, recall, F-beta, and PR-AUC per lot and per line, not a single micro-average.
- Reserve a calibration split, grouped by board, exclusively for choosing the operating threshold.
The key judgement is that leakage inflates the metric and also distorts threshold selection. Fixing the split may lower the reported number, but it is the only number that predicts the new line.
3. Recommend an intervention plan in priority order under the 20 ms constraint and the 40:1 cost ratio.
Read the worked answer
Given a 40:1 cost ratio, recall is worth roughly 40 times more than precision. The F-beta that reflects this uses beta = sqrt(40) ≈ 6.3, so F6.3 or an explicit expected-cost calculation is the right objective.
Priority order:
- Threshold tuning on the grouped calibration set. This costs nothing at inference and can raise recall immediately. Choose the threshold that minimizes expected cost: 40 × false negatives + 1 × false positives. Report the resulting precision so operations understands the alarm load.
- Cost-sensitive loss. Replace plain cross-entropy with class weights or focal loss so the minority class contributes meaningfully. This changes training only and keeps the 20 ms inference budget intact.
- Resampling or hard-negative mining if weighted loss still ranks defects poorly.
- Collect and label new-line images before launch; a model calibrated on old lighting will drift.
- Architecture change last, because it threatens the latency budget.
The judgement is to exhaust zero-latency levers first. Dropout and extra epochs are not on this list because they do not address imbalance or leakage.