Predictive Maintenance Threshold Decision After Sensor Firmware Change
A chemical plant operates 40 centrifugal pumps. A predictive-maintenance model scores vibration, temperature and pressure data every 15 minutes and flags failures 72 hours ahead. At the current threshold of 0.80, validation showed 94% recall and 61% precision; operations says the resulting 30 extra inspections per month consume two technician shifts. Three pumps had sensors replaced last quarter, and the vendor changed the vibration sensor firmware, shifting the signal baseline. The model was last retrained nine months ago on 14 months of data. Unplanned pump failures cost about USD 45,000 per event, while a planned inspection costs USD 1,200. The plant cannot take pumps offline outside a monthly maintenance window. A recent failure on pump P-17 was scored 0.42 eight hours before breakdown. The reliability manager wants to lower the threshold to 0.65 to catch more failures; the finance controller wants to raise it to 0.90 to cut inspections.
1. Should the plant lower the threshold to 0.65, raise it to 0.90, or take another action? Justify your recommendation with the cost and operational constraints.
Read the worked answer
The threshold is not the first problem: the sensor firmware change and the P-17 miss at 0.42 point to distribution shift or a feature-calibration fault. Lowering to 0.65 would increase recall only if the score distribution still ranks failures correctly; it would also increase false inspections, and two technician shifts may not absorb the extra volume. Raising to 0.90 cuts workload but raises missed-failure cost and safety risk; one missed event at USD 45,000 exceeds roughly 37 planned inspections at USD 1,200. A defensible sequence is: (1) quarantine the affected sensor data and recalibrate or retrain on post-firmware data; (2) validate a new threshold on a time-based holdout using expected cost = false negatives x 45,000 + false positives x 1,200, subject to a capacity cap of about 40 inspections per month; (3) if capacity remains, choose a threshold below 0.80 only after recalibration, not 0.65 blindly; (4) keep human review for pumps with high failure consequence. The finance controller is right that false positives matter, but the reliability manager is right that the current model missed a known failure. Fix the model before trading recall for inspections.
2. Define the monitoring and retraining triggers you would put in place, including leading and lagging indicators.
Read the worked answer
Monitor data quality and drift by pump and sensor: missing readings, vibration RMS mean and variance, temperature and pressure distributions, and firmware version. Use a drift detector on features and compare against the training baseline weekly; any sensor replacement or firmware change is a forced review, not a routine metric. Leading indicators: percentage of scores near threshold, inspection confirmation rate, false-alarm rate, and number of pumps with abnormal data. Lagging indicators: missed failures, unplanned downtime, recall and precision from confirmed outcomes, and cost per avoided failure. Retraining triggers should be explicit: firmware or sensor change; drift above a pre-agreed statistical threshold, for example population stability index above 0.2 on key features; precision below 50% or recall below 90% on the last 90 days of confirmed events; two missed failures within one quarter; or a change in maintenance capacity. Before retraining, freeze a labeled holdout and compare champion versus challenger in shadow mode for two maintenance cycles. Do not retrain only on inspection-confirmed failures without accounting for uninspected pumps, or the model will learn from a biased label set.
3. How would you validate the model after the firmware change before returning it to full production?
Read the worked answer
Use a time-based split that puts all post-firmware data in the test set and keeps pre-firmware data for training or comparison. Stratify by pump, sensor type, and failure mode; confirm that the P-17 case and similar near-misses are represented. Compare three candidates: current model unchanged, recalibrated model, and retrained model. Report recall at the operating threshold, precision, calibration, and expected cost, not just AUC. Run a shadow-mode period through at least two monthly maintenance windows so that predictions can be checked against inspections. Require sign-off from reliability engineering and operations, and document the firmware version as a model input or exclusion. If performance is unstable, keep the old model with a conservative threshold and manual review for affected pumps until enough post-change labeled data exists.