Two Suppliers, One p-Value
Pinnacle Plastics sources resin from two suppliers. A key property, tensile strength, must be at least 38 MPa. Samples: Supplier A, n = 30, mean 42.1 MPa, s = 3.8; Supplier B, n = 30, mean 40.2 MPa, s = 6.1. A pooled two-sample t-test gives t = 1.45, p = 0.15. Procurement concludes the suppliers are equivalent and intends to award the contract to B, which is 9% cheaper. Two facts surface later: every Supplier A sample came from a single production line, while Supplier B samples came from two lines running different catalyst grades; and both suppliers' samples came from a single week of shipments. The team's statistician estimates the study had only about 40% power to detect a 2 MPa difference.
1. What is wrong with the conclusion that the two suppliers are equivalent?
Read the worked answer
Failing to reject a null hypothesis is not evidence of equivalence - and the mean is the wrong statistic for a one-sided specification.
- The metric answers a different question than the decision does. Procurement must decide which supplier reliably delivers ≥ 38 MPa, not whether the means differ. With a one-sided spec, the tails matter:
- Supplier A: z = (38 − 42.1)/3.8 = −1.08 → ≈ 14.0% below spec (~140,000 ppm)
- Supplier B: z = (38 − 40.2)/6.1 = −0.36 → ≈ 35.9% below spec (~359,000 ppm) Supplier B's larger spread more than offsets its slightly lower mean. Whatever the p-value, B is expected to produce roughly 2.5× the out-of-spec material.
- Low power plus non-significance equals no information. At ~40% power, a 2 MPa difference is genuinely detectable only 4 times in 10. "Not significant" here means "the study could not have seen it."
- Equivalence requires an equivalence test. If the real question is "are they the same," use a TOST/confidence-interval approach against a pre-stated equivalence margin - not a failure to reject.
- Capability check: (42.1 − 38)/(3×3.8) ≈ 0.36 and (40.2 − 38)/(3×6.1) ≈ 0.12 - both are far below any acceptable Cpk, which itself should trigger a broader escalation, not a supplier award.
(Note: with equal n, the pooled and Welch tests give nearly identical t and p, so unequal variances are not the flaw here. The flaw is the question being asked.)
2. What confounding and sampling issues must be resolved before any award decision?
Read the worked answer
The apparent supplier effect may be a line, catalyst, or time effect wearing a supplier label.
- Confounding with production line: Supplier B's two lines run different catalyst grades. If the strengths differ by line, B's distribution is a mixture - the s = 6.1 may be an artifact of blending two populations, not inherent supplier variability. A histogram showing two modes is the tell; a single s hides it.
- Nested structure: Part variation is nested within lot within line within supplier. Treating all 30 samples as independent overstates the effective sample size and understates the standard error.
- Time window: One week of shipments from each supplier samples one set of lots, one set of operators, one set of raw-material deliveries. It cannot represent annual supply.
- Measurement comparability: Was the same lab, sample preparation, machine, and operator used for both suppliers? If not, the between-supplier difference is partly between-test-conditions.
Recommended analysis plan:
- Stratify - plot and summarize Supplier B by line; run a two-sample comparison A vs. each B line, and a test for B-line differences (means and variances, e.g., Levene's test for spread).
- Verify distribution shape and normality before trusting any estimate of the fraction below spec; with a possible mixture, empirical PPM from the data may be preferable to z-based extrapolation.
- Re-sample with a proper plan: multiple lots spread over weeks, random selection within lots, identical test conditions, measurement system verified first.
- Then compare on capability (expected PPM below spec) and total cost - the 9% price advantage must be weighed against the incremental cost of the extra ~22% of material below spec, including scrap, rework, sorting, and returns.
3. A manager proposes running 300 samples per supplier to "make the p-value significant." How should the team respond?
Read the worked answer
More of the same data will buy a smaller p-value and very little decision quality.
- Significance is assured, not informative. With a fixed 1.9 MPa difference and the observed spreads, the standard error shrinks with √n; at n ≈ 300 the difference becomes "significant" almost mechanically. But the difference was already known; the question was never whether the suppliers differ slightly in mean.
- The real issue is not resolved by n. Non-representative sampling (one week of shipments), confounded lines and catalyst grades, and unverified measurement comparability are bias problems. Bias does not shrink with sample size - a large biased sample produces a precise wrong answer. Increasing n from a single week only estimates that week more tightly.
- It also ignores practical significance. A 1.9 MPa mean difference with a 6.1 vs. 3.8 spread has clear engineering consequences at a 38 MPa spec limit, so the project needs a capability and PPM comparison, not a p-value threshold.
Better response:
- If the business question is truly "are they equivalent," pre-specify an equivalence margin and run a TOST/CI procedure - n is then chosen to give adequate power for that margin, not to chase a p-value.
- If the question is "which supplier meets the spec reliably," choose n to estimate PPM below 38 MPa with acceptable confidence, and fix the sampling frame first (lots × weeks × lines).
- Verify the measurement system, replicate within lots to separate lot-to-lot from within-lot variation, and stratify the analysis. Sample size last, design first.