Overfitting a Defect Classifier on an Imbalanced Line
A factory deploys a ResNet-50 defect classifier on 9,600 labelled images across eight defect classes; the rarest class has 240 images and the largest 3,100. Training uses Adam at 1e-3, batch 32, 60 epochs, and only horizontal flips. Training accuracy reaches 99.1% and validation accuracy 78.4%; training loss falls to 0.02 while validation loss bottoms out at 0.55 around epoch 22 and rises to 0.61 by epoch 60. Validation macro-F1 is 0.71 and recall on the rare class is 0.38. A benchmark shows the model already meets the line's 15 fps requirement at 224×224, so latency is not the constraint. The team proposes doubling the network width and training for 120 epochs to let it learn more features. The line runs 1,200 parts per hour.
1. Diagnose the training curves and evaluate the proposal to double the network width.
Read the worked answer
The curves show overfitting, not underfitting. Training loss is 0.02 while validation loss is 0.55–0.61, and training accuracy exceeds validation accuracy by more than 20 points. The validation-loss minimum at epoch 22 marks the point where generalization stopped improving; the remaining 38 epochs added capacity to memorize training images.
Doubling the width and training longer would increase capacity and almost certainly widen the gap. The first actions are to restore the epoch-22 checkpoint and add early stopping, then strengthen regularization and augmentation: random crops, rotation, contrast and brightness changes, weight decay, and dropout. A smaller backbone or a lower input resolution is worth testing if accuracy remains short.
Only after regularization should width be tested, and then as a controlled ablation with the same early-stopping rule. The 15 fps benchmark shows the current model fits the latency budget, so there is no speed reason to enlarge it.
2. Rare-class recall is 0.38. What should the team change, and which metrics should gate the next release?
Read the worked answer
Macro-F1 of 0.71 hides the failure: a defect class with 240 images has only 38% recall, meaning most rare defects pass. Start with class-weighted loss or focal loss, and use stratified splits so every fold contains rare examples. Oversample the rare class with augmentation rather than duplicating images. Tune per-class thresholds instead of using a single 0.5 cut-off.
Inspect the confusion matrix before adding complexity. If rare defects are confused with a visually similar class, targeted labelling and crops of the confusing region help more than a new architecture. If they are missed entirely, add harder negative mining.
Gate the next release on per-class recall and macro-F1, not overall accuracy. A practical target is rare-class recall above 0.85 with false positives inside the rework capacity. Report the full confusion matrix and the confidence interval, since 240 images give noisy estimates.
3. The line runs 1,200 parts per hour. Should the current model be deployed while improvements continue?
Read the worked answer
Convert the rate into a decision context: 1,200 parts per hour is 20 parts per minute, so a 38% miss rate on rare defects means several escapes per hour if the rare class occurs at even a modest rate. Whether that is acceptable depends on the cost of an escape versus the cost of manual inspection, but a safety-critical defect should not be released at 38% recall.
A staged deployment is reasonable only with a human fallback for the rare class. Run the model in parallel with inspection, route low-confidence and rare-class predictions to a person, and collect the resulting labels. This protects quality while generating the data needed to improve recall.
Set an exit condition: remove the human fallback for a class only after its recall exceeds 0.85 on a held-out, time-separated set and false positives stay within rework capacity. Deploying the current model alone would trade a scheduling convenience for unmeasured quality risk.