Predictive Maintenance Model Built on Unreliable Sensor Data

The claim is not measurable as stated. 'Accuracy' without a stated base rate, a confusion matrix, or a defined positive event cannot be interpreted. Reconstruct what the data can support:

• Alerts raised: 47. Investigated: 31, of which 7 were true defect progressions and 24 were spurious (mostly sensor faults). Precision among investigated alerts = 7/31 = 23%. Almost four in five investigated alerts consumed maintenance effort for no defect.
• Sixteen alerts were never investigated, so precision across all 47 lies somewhere between 7/47 = 15% and 23/47 = 49% - an unhelpfully wide range. The organisation cannot yet state its own alert quality.
• The one failure without warning is the most serious data point. Any credible system must be judged on missed events as well as false ones, and a single missed gearbox is a serious counter-example when the population is 14 machines over six months.
• Detection 'four weeks ahead' is meaningless here because the data has no timestamps of defect initiation - the seven true positives were never characterised against lead time.

What the evidence supports: the model is detecting something correlated with degradation in roughly a quarter of the cases where it fires, and it is not yet reliable enough to be the sole trigger for intrusive maintenance. It is potentially useful as a screening tool layered on top of the existing regime.

What to tell the manager:

1. 'Ninety-two per cent' cannot be validated and should not be quoted to the board. Present 23% precision on investigated alerts plus 16 unknowns instead; that is defensible and it points directly at the fix.
2. Do not switch to model-driven scheduling next quarter. A 23% precision rate on a calendar-based overhaul programme would trade a predictable cost for an unpredictable one and undermine confidence in the tool.
3. Set an explicit target for the next two quarters: for example, investigate 100% of alerts, drive precision above 60%, and demonstrate a documented lead time of at least two weeks on the true positives. Then revisit the scheduling change.
4. The missed gearbox should be reviewed in detail - the slow trend that nobody saw suggests an alerting or interface problem as much as a model problem.

Diagnosis - the model is being blamed for a data problem:

• Stale sensors. Two vibration points reading a flat 0.1 mm/s for eight months is physically implausible on a running machine. A constant value is the classic signature of a dead transducer or a frozen value cached by the historian. Those tags are providing no information, yet the model treats them as healthy steady state - which is exactly how the missed gearbox becomes possible.
• Unit and scaling inconsistency. Three tags duplicated in µm and mm/s means the model may be comparing quantities that differ by a factor of 1,000. Duplicated tags with divergent scaling are the most dangerous class of error because both look plausible individually.
• Backfill with last known value. This fabricates data: gaps become flat lines, which the model reads as stability. It also inflates apparent uptime and destroys the statistical properties the model was trained on. Gaps must be flagged as gaps.
• Sampling adequacy. 1 Hz sampling of overall RMS only discards the spectral content that distinguishes bearing defect frequencies from unbalance, misalignment or a resonating structure. The model can only learn what the data contains.

Remediation sequence (do not tune the model first):

1. Tag inventory and reconciliation: build a register of all model inputs, their units, scaling, ranges and source instruments; resolve duplicates to a single authoritative tag; document the mapping.
2. Validity rules in the historian or an intermediate layer: range checks, rate-of-change checks, and a 'stuck value' detector that flags any tag unchanged beyond tolerance for more than a defined period. Route flagged tags to a data-quality dashboard with an owner.
3. Replace last-value backfill with explicit nulls plus a quality flag, and exclude flagged periods from model training and scoring.
4. Repair or replace the dead sensors and add them to a calibration and verification schedule.
5. Improve the input data where it pays: add spectral features (envelope or band energies) on the highest-criticality machines rather than more machines with poor data.
6. Only then retrain and back-test on the cleaned history, using a time-ordered split so the model is never trained on data from the period it is tested on.

Frame the choice honestly. The options are not 'AI scheduling' versus 'calendar overhauls'. The realistic options are (a) calendar-only, which over-maintains healthy machines and still misses the failures that develop fast; (b) alert-driven with 23% precision, which is what is currently happening and which is eroding confidence; or (c) a risk-ranked hybrid in which the model screens and the calendar covers what the model cannot yet see.

Recommendation: a hybrid, with the model's authority explicitly limited.

1. Rank by criticality, not by alert count. Classify the 14 machines by failure consequence (production loss, safety, repair cost). Route model alerts on the top four or five to investigation within a stated response time; route alerts on the remainder to the planner's routine review. This converts an unmanageable 47 alerts into a manageable handful.
2. Give the planner capacity by design. Budget a fixed number of investigation hours per week (say four) and a standard investigation protocol - 30-minute route check with a handheld analyser - so each alert costs a predictable amount rather than an open-ended job. A 30-minute check that clears 24 false alerts is a much cheaper outcome than an unplanned gearbox failure.
3. Keep calendar overhauls for now, on the machines where they are cheap and the consequence is low, and where the model has no proven track record. Then agree a sunset test: if the model achieves its precision and lead-time targets for two consecutive quarters on a machine, its calendar overhaul interval is extended by one step. That gives the reliability engineer a defensible, evidence-based exit rather than an argument.
4. Fix the alert-to-action path. The missed gearbox trend was visible and nobody saw it. Alerts need a named owner, a due date, a status field, and a weekly review of open items - otherwise the system is generating information that the organisation cannot absorb.
5. Report the outcome measures that both parties care about: alerts investigated as a percentage of alerts raised, precision, unplanned failures on covered machines, maintenance hours per machine, and downtime. If model-driven scheduling is genuinely better, those numbers will show it within two quarters; if it is not, the same numbers justify stopping.

What not to do: do not declare the pilot a success on the vendor's metric, and do not extend it to more machines before precision and the alert-to-action loop are fixed. Scaling a 23%-precision system multiplies the planner's problem.