Second-Source Filament: A Non-Significant Result Is Not an Approval
A procurement-led project is evaluating a second source for nylon filament used in a safety strap. The drawing requires tensile strength of at least 45 N. Supplier A is the incumbent; supplier B is 12% cheaper. The Green Belt runs a two-sample t-test on 8 spools from each supplier: A mean 47.2 N, s = 2.1; B mean 45.4 N, s = 2.6. The test gives p = 0.15 at α = 0.05, and the 95% confidence interval for the difference (A − B) is −0.7 to 4.3 N. The sponsor circulates a note: 'Statistically, B is the same as A - switch and save the money.' Purchasing has already drafted the change notice. Historical process data show the between-spool standard deviation is about 2.4 N. The strap has no 100% downstream test; failures are detected at final functional test on a sample basis.
1. Is the sponsor's conclusion supported by the test? Quantify what the study could and could not detect.
Read the worked answer
No. A p-value of 0.15 means the study failed to reject the null hypothesis; it is not evidence that the two suppliers are equivalent. Absence of evidence is not evidence of absence.
Quantify the blindness of the test. With n = 8 per group and σ ≈ 2.4 N, the standard error of the difference is 2.4 × √(2/8) ≈ 1.20 N. At 80% power and α = 0.05 (two-sided), the smallest detectable difference is about 2.8 × 1.20 ≈ 3.4 N. The observed gap is only 1.8 N - well inside the noise band. The study was never capable of resolving a difference of the size that matters.
The confidence interval says the same thing: −0.7 to 4.3 N is compatible with B being slightly worse than A, equal to A, or 4.3 N better. A range that wide cannot support a sourcing decision on a safety-critical part.
Worse, the test answers the wrong question. The decision is not 'is B different from A?' but 'is B capable against the 45 N specification?' Without a significance test, B's estimated defect rate below 45 N is roughly 44% (z = (45 − 45.4)/2.6 = −0.15); for A it is about 15% (z = −1.05). Those are not interchangeable processes.
2. Recommend a decision and specify the evidence needed to make it. State what would change your recommendation.
Read the worked answer
Do not sign the change notice on this study. Recommend a hold on supplier B while the evidence is rebuilt.
Evidence needed:
- Reframe the question as capability, not comparison. For each supplier, collect at least 100 measurements spanning several lots and estimate the fraction below 45 N with a confidence interval. Even supplier A, at mean 47.2 and s = 2.1, has Cpk ≈ (47.2 − 45)/(3 × 2.1) ≈ 0.35 - nowhere near the 1.33 usually required for a safety strap. That finding may be more important than the sourcing question.
- Size the comparison properly. To detect a 1.8 N difference with σ = 2.4 at 80% power: n = 2 × (2.8 × 2.4 / 1.8)² ≈ 28 per group, so run about 30 spools per supplier, randomized across lots.
- Separate lot-to-lot from within-lot variation using a variance-components study; a supplier that is fine on average but erratic between lots is a risk on an untested safety part.
What would change the recommendation: a properly powered study showing B non-inferior to A with a lower confidence bound above a pre-agreed margin tied to the 45 N limit, plus acceptable B capability and lot-to-lot stability. Cost savings do not override an unquantified safety risk.