Northline Components: Failure Prediction with 60 Labels and a 99.4% Number
Northline Components runs 40 CNC machines across two plants. Maintenance logs record roughly 60 unplanned failures over the past 18 months, tagged by machine and date, but tags are hand-entered, sometimes several days late, and 9 of the 40 machines have no failure records at all. Sensors sample vibration, temperature, and spindle load every 10 seconds. Leadership wants a model that outputs 'failure expected within 72 hours' and has budget for exactly one build-and-evaluate cycle before the next board review. A vendor demo showed 99.4% accuracy on a similar dataset, and the engineering lead wants to match that number before proposing anything else. The maintenance team can realistically act on about 15 alerts per week.
1. The team proposes headline accuracy as the primary success metric. Using the numbers in the setup, show why this is a trap and state what you would report instead.
Read the worked answer
The base rate makes accuracy meaningless here.
- 40 machines x ~540 days = ~21,600 machine-days of observation.
- 60 failures across those machine-days = roughly a 0.28% positive rate.
- A model that always says 'no failure' scores about 99.7% accuracy while being useless.
So the vendor's 99.4% is worse than a trivial classifier. Any accuracy figure near 99% on this dataset is evidence the model is predicting the majority class, not evidence of skill.
What to report instead
- Operating-point metrics tied to capacity: precision and recall at the alert budget the team can absorb (15 alerts/week, ~780/year). At a 0.28% base rate, even a good model produces mostly false alarms, so precision-at-k is the number maintenance actually feels.
- PR-AUC rather than ROC-AUC, because ROC-AUC looks flattering under heavy imbalance.
- Cost-weighted recall: a missed failure has a downtime cost; a false alarm costs an inspection. State both and pick a threshold against them.
- Lead time distribution: median hours between first alert and failure, since 72 hours is the actionable window.
Accuracy can be listed as a footnote; it should not be the acceptance criterion.
2. Given the late hand-entered tags and the 9 machines with no failure history, decide whether to train a supervised model in this one cycle, and justify the architecture you would actually build.
Read the worked answer
Do not build a pure supervised failure classifier in one cycle. The label set is small (60 events), temporally noisy, and structurally incomplete: 9 machines contribute zero positives, so a supervised model cannot learn anything for them and any random split will let the model memorise machine identity.
Recommended hybrid, delivered in one cycle
- Unsupervised anomaly detection per machine: learn each machine's normal vibration/temperature/load envelope from its own history. This works on all 40 machines, including the 9 with no failures, and needs no labels.
- Supervised layer only where labels exist, using a wide label window (e.g. a failure tag applies to +/-3 days) to absorb late entry, and discarding tags the maintainers cannot date to within that window.
- Rank, don't classify: output a ranked alert list per week so the 15-alert budget is respected, and let maintainers label the outcome of each alert. That feedback loop is the real training data for cycle two.
Explicit non-goals in cycle one: no deep sequence model, no cross-machine transfer learning, no 72-hour probability claims. The honest deliverable is 'top-15 weekly ranked anomalies with a measured hit rate,' which is also what earns the second iteration.
3. Describe the evaluation split and validation design that would prevent an optimistic result from reaching the board.
Read the worked answer
Two leakage risks dominate: time and machine identity.
- Never split randomly. Sensor rows 10 seconds apart are near-duplicates; a random row split puts the same machine's same failure event in both train and test, inflating every metric.
- Split by time. Train on the first ~12 months, validate on the most recent ~6. Failure regimes drift (tooling changes, seasonal load), so the recent window is the only honest estimate.
- Group by machine within any cross-validation so a machine appears in exactly one fold; otherwise the model learns 'machine 17 is the noisy one' rather than a failure precursor.
Validation design
- Freeze the split before any modelling and record it.
- Hold out the 9 zero-failure machines entirely for unsupervised validation - do they generate a plausible alert volume, or a flood?
- Report a paired comparison against the current threshold-based rule, on the same weeks, so the board sees incremental value rather than an absolute score.
- State the confidence interval. With 60 events total and maybe 20 in the test window, metric uncertainty is large; a point estimate presented alone is misleading.