Reading a Vendor White Paper Before an Investment Decision
A maintenance manager forwards a 14-page white paper from a predictive-maintenance platform vendor. The headline claim is: 'Customers achieve 40 % fewer unplanned stoppages within six months.' The evidence section cites one plant with 12 injection-moulding machines and six months of use. Downtime hours are taken from the vendor's own dashboard. During month 1 the plant also replaced its CMMS and introduced a new shift-handover routine; the paper does not discuss this. In month 1 a stoppage counted as unplanned if it lasted longer than 15 minutes; from month 3 the threshold was 30 minutes. No comparison or companion plant is mentioned. Six of the twelve machines received new heater bands in month 2. The platform's alarm thresholds were unchanged throughout. You must write the technical review for an investment decision, in English, for a mixed audience of engineers and finance staff.
1. Identify the three strongest threats to the validity of the 40 % figure, rank them, and justify the ranking.
Read the worked answer
Ranked threats
1. The change in the downtime definition (strongest). Moving the counting threshold from 15 to 30 minutes removes every stoppage between those two values from the record. That reduces the headline number arithmetically, before any predictive-maintenance effect exists. In moulding plants the 15–30 minute band is typically a large share of interventions, so this alone could account for a substantial part of the 40 %. It is ranked first because it does not depend on any assumption about behaviour: the result is manufactured by the method.
2. Concurrent changes in the plant (second). The CMMS replacement, the new shift-handover routine and 12 new heater bands (on half the sample) all plausibly reduce unplanned stoppages. None is isolated from the platform's effect, and with n = 12 there is no way to separate them. Ranked second because these effects are real but uncertain in size and direction.
3. Self-measurement by the vendor with no control group (third). Downtime is read from the vendor's own dashboard, and there is no comparison plant. This is a bias risk rather than a demonstrated distortion: it could favour the vendor, but nothing in the document shows it did. Ranked third for that reason - but it means the figure is not independently verifiable at all.
Also worth naming: single site, short window, no stated baseline period, and the 12 machines are not described as representative.
2. Distinguish the claim from the evidence and rewrite the headline in a form you can defend in the review.
Read the worked answer
Claim versus evidence
| Claim | What the document actually shows |
|---|---|
| 'Customers' - a generalised population | 12 machines at one plant, one site |
| '40 % fewer' - a precise causal effect | A percentage change in a metric that was redefined mid-trial, with several concurrent interventions |
| 'within six months' - a time-bounded result attributable to the platform | A six-month observation window containing an unmeasured mixture of causes |
The claim is causal, quantified and generalised. The evidence is single-site, self-reported, confounded and definitionally unstable. The gap between them is the review's central finding.
Defensible rewrite
In a six-month pilot on twelve machines at one plant, the vendor reports a reduction in recorded unplanned downtime. Because the downtime definition changed during the pilot and several maintenance processes were altered at the same time, the share of any reduction that can be attributed to the platform cannot be determined from this document.
Hedging devices used, and why
- the vendor reports - attribution verb; the reviewer is not asserting the result.
- recorded downtime - reminds the reader the metric is a measurement artefact, not the physical stoppages.
- cannot be determined from this document - a bounded negative claim, which is safer than an accusation.
- No number is repeated, because repeating '40 %' even in a critique tends to fix it in the reader's memory.
3. State what evidence would make the claim defensible, and explain in one sentence why finance staff in particular need that answer.
Read the worked answer
What would make it defensible
- A fixed metric for the whole period, agreed before the trial starts and audited afterwards - the same stoppage threshold in month 1 and month 6.
- A baseline of at least twelve months so seasonal and product-mix effects are visible.
- A comparison line or plant running the same CMMS and handover routine without the platform.
- An independent data source - the plant's CMMS or production records rather than the vendor's dashboard.
- A documented list of concurrent changes, with dates, so their contribution can at least be bounded.
- Machine-level results, not a pooled average, so the variance and the number of machines actually helped are visible.
For the finance audience The one sentence to add to the review:
The relevant question for the business case is not whether downtime fell during the pilot - several changes could have caused that - but how much of the fall would disappear if the platform were removed, and the white paper contains no evidence that can answer it.
Why that sentence matters Finance is being asked to buy a recurring licence on the strength of a benefit that has not been isolated. Framing the gap as an attribution problem, not as a credibility attack on the vendor, keeps the meeting on the decision rather than on the document.