Rare-defect image classifier tuning
A semiconductor manufacturer trains an image classifier to detect 18 defect classes on production-line images. The dataset has 240,000 labeled images; the rarest class has 380 examples, the most common 61,000. Images come from 14 cameras with different lighting, and labels come from two vendor teams with 87% agreement. Current practice: a data scientist manually edits a notebook, tries 8 learning rates, selects the run with the best overall accuracy (0.94), and promotes it. In production, recall for the rarest class is 0.31, and each missed defect costs $4,200 per wafer lot. Each training run takes 9 hours on a GPU cluster shared with other teams. Auditors ask why the selected model was chosen, and the team cannot explain the tuning process.
1. Redesign the experimentation and tuning strategy to improve the rare class without wasting expensive GPU time.
Read the worked answer
Replace manual notebook edits with tracked experiments. Define a time- and camera-based split so validation resembles future production and rare classes appear. Log the code commit, data version, split, augmentation, class weights, and metrics for every run. Use a hyperparameter sweep with a bounded search space for learning rate, augmentation strength, class weight, and image size, plus an early-termination policy such as median stopping or bandit. Cap trials and use a smaller proxy subset for screening, then retrain promising configurations on full data. That saves GPU because 9-hour full runs are not spent on obviously poor trials. Select by cost-weighted recall or macro recall, not overall accuracy, which is misleading with 0.4% rare-class prevalence. Keep the sweep configuration and selection rule so auditors can see why the model was chosen.
2. Recommend metrics and imbalance-handling techniques for this cost structure.
Read the worked answer
Use per-class recall, precision, confusion matrices, and a cost matrix: a missed rare defect costs $4,200 per wafer lot. Consider class-weighted loss or focal loss, targeted oversampling of rare classes only in training folds, and augmentation. Do not oversample before splitting. Calibrate thresholds per class to the cost tradeoff rather than using a default 0.5 threshold. Evaluate per camera and lighting condition because the 14 cameras create different distributions. Report confidence intervals for rare-class recall because 380 examples is a small sample. Overall accuracy should be a secondary reporting metric only.
3. How should the team address label noise and camera variation, and what evidence must be retained for audit?
Read the worked answer
Address label noise by adjudicating a sample, tracking label quality by vendor, and possibly using confident learning; do not silently relabel. Normalize and augment per camera, and add camera ID only if it will be available at inference. For audit, store the sweep configuration, selection rule, all trial metrics, final model card, and approval record. The explanation for the selected model should be, for example, best cost-weighted validation recall subject to latency, not best accuracy. Retain the dataset version and split definition so the result can be reproduced.