Choosing a Classifier for Rare Machine Failures
Atlas Components runs a CNC machining line with 14 vibration, temperature and pressure sensors. Maintenance engineers have labelled 1,800 run-to-failure windows over three years; 6.8% of windows end in failure within 24 hours. Forty-two engineered features summarise recent sensor behaviour. The team compared four classifiers on a random 80/20 split. Logistic regression reached 0.74 ROC-AUC, random forest 0.81 with 450 ms inference on the edge gateway, gradient boosting 0.83 with 120 ms, and a small multilayer perceptron 0.79 but with fold scores ranging from 0.61 to 0.88. The gateway budget is 200 ms per prediction and the line produces about 900 prediction requests per shift. Maintenance can handle roughly three false alarms per day, but missing a failure costs about EUR 18,000 in downtime. The engineers want a recommendation, not another benchmark table, and they ask whether a deep model should be used because it is more modern.
1. Recommend a classifier for the edge gateway. Compare at least three options using the evidence, not just AUC.
Read the worked answer
On the evidence, gradient boosting is the best default: highest ROC-AUC at 0.83 and 120 ms inference, inside the 200 ms budget. Random forest is close on AUC at 0.81 but 450 ms breaches the gateway budget, so it would need optimisation before consideration. Logistic regression is the most interpretable and fastest, but 0.74 AUC may be too weak given the EUR 18,000 cost of a missed failure; it remains a useful baseline. The multilayer perceptron should not be selected: 0.79 AUC with folds from 0.61 to 0.88 indicates high variance, and 1,800 windows is small for a deep model. AUC alone is insufficient because the operating point matters; compare precision-recall curves at the alarm budget. I would pilot gradient boosting, with logistic regression as a challenger and a documented threshold, rather than choose on modernity.
2. Explain how you would validate the chosen model and set the alarm threshold. Include the role of the false-alarm budget and the cost of a missed failure.
Read the worked answer
Use grouped, time-ordered validation: hold out entire machines and later time periods so the model cannot memorise a machine's signature. Report PR-AUC and recall at the false-alarm budget, not just ROC-AUC. Translate the budget: about three false alarms per day; if the line makes 900 requests per shift across, say, two shifts, that is a low false-positive rate, so the threshold must be set from the validation precision-recall curve and the estimated alert volume. Include the EUR 18,000 missed-failure cost and the maintenance inspection cost in a simple cost matrix. Calibrate probabilities if the threshold is expressed as risk. Test on the most recent six months, then run a shadow period. The threshold should be reviewed when sensor calibration, production mix or maintenance policy changes.
3. The engineers ask whether a deep model should be preferred because it is more modern. How do you answer, and what would change your recommendation?
Read the worked answer
I would answer that modern is not a selection criterion; validation performance, stability and operating cost are. With 1,800 labelled windows and 42 engineered features, a deep model has little data per parameter and shows unstable fold performance. The gradient-boosted model already meets latency and has better AUC. A deep model could become preferable if the team collected much more data, used raw high-frequency sensor sequences, or needed multi-sensor temporal patterns that engineered features miss. Even then, I would require a pre-registered comparison on grouped time-based splits, latency testing on the actual gateway, and a fallback. The recommendation would change if the MLP's variance narrowed with more data and it beat gradient boosting at the fixed alarm budget, not merely on average AUC.