Fabrikam's CNC Failure Prediction Model
Fabrikam Components runs 40 CNC machines. Sensors log vibration, temperature and spindle load every 30 seconds, and twelve months of history is available. Unplanned failures occurred 61 times in that period - roughly 0.8% of machine-hours. A missed failure costs about $40,000 in lost production; a false alarm costs about $300 for an unnecessary inspection. Maintenance planners need at least four hours' notice to act. The data team plans to use Automated ML in Azure Machine Learning, split the data randomly 80/20, and select the model with the highest accuracy. Their first run reports 99.4% accuracy on the test set and the team wants to ship it.
1. Which machine learning problem type is this, and why is 99.4% accuracy a misleading headline number?
Read the worked answer
It is binary classification - each observation is predicted as 'failure within the horizon' or 'no failure'. It is not regression (we do not need a time-to-failure number, we need an alert) and not clustering (the outcome label exists).
The accuracy figure is misleading because of class imbalance. Only about 0.8% of machine-hours are positive, so a trivial model that always predicts 'no failure' scores about 99.2% accuracy while catching zero failures. The reported 99.4% is barely distinguishable from that baseline, and may be worse than it once you account for the specific test rows.
Accuracy is the wrong selection metric whenever the classes are this skewed and the costs are this asymmetric. The judgement point: a metric that looks excellent can be actively dangerous, because it invites you to ship a model whose only skill is predicting the majority class. Automated ML will happily rank models by accuracy if you tell it to - the metric is a configuration decision, not an objective fact.
2. Which metric should drive model selection, and how should the data actually be split?
Read the worked answer
Metric. Optimise for recall on the failure class at an acceptable precision, because the cost asymmetry is roughly 130:1 ($40,000 versus $300). Concretely: select on AUC / precision-recall curve area to rank candidate models, then choose the operating threshold by cost, not by default 0.5. Report recall, precision and the expected cost per month rather than a single accuracy number.
Split. A random 80/20 split is wrong here and is the real bug. Consecutive 30-second readings from the same machine are near-duplicates, so a random split puts rows from the same failure episode in both train and test. The model effectively memorises the episode and the test score is leakage, not skill.
Use a time-based split: train on the first 9-10 months, validate on the next, test on the final month. Optionally group by machine as well. This mirrors deployment, where you always predict forward in time, and it will produce a much lower and much more honest score - expect that, and do not treat the drop as a regression.
3. Planners need four hours' warning. How do you turn a continuous stream of scores into alerts that actually meet that requirement?
Read the worked answer
Three linked changes, each of which the raw model does not give you:
- Label the horizon, not the instant. A row is positive if a failure occurs within the next four hours (eight readings at 30-second cadence, or use a rolling window of aggregates). Without this, the model learns to fire only at the moment of failure - accurate, useless.
- Threshold by cost. Set the alert threshold below 0.5 so that the 130:1 cost ratio is reflected. Compute expected monthly cost as recall-weighted downtime versus false-alarm volume, and show that trade-off explicitly to planners, because they will be the ones doing the unnecessary inspections.
- Add de-duplication. A threshold crossed on every reading will emit dozens of alerts per episode. Suppress repeats per machine per window and require a minimum persistence (e.g. two consecutive readings) before alerting.
Also schedule retraining and drift monitoring: a model trained on a year of data will degrade as machines are serviced, and a silent precision collapse produces alert fatigue, which is how these systems get switched off.