A Supplier Conclusion Confounded by Reactor, Season, and Temperature
A specialty chemical reactor produces a batch yield averaging 82%. The team compared two raw-material suppliers: 14 batches from supplier A averaged 84.1% and 12 from supplier B averaged 80.6%; a two-sample t-test gave p = 0.03. The team concluded supplier A is superior and proposed switching. A reviewer notices that supplier A batches were run mostly in Reactor 2 during summer, while supplier B batches were run in Reactor 1 during winter. Reactor 2 has a newer jacket and better temperature control. Batch temperature setpoints also differed: A averaged 3 degrees Celsius higher. The team ran 11 additional comparisons on other variables and found two more p-values below 0.05. Sample sizes per group are small, and variances differ. The plant manager wants to switch suppliers next month.
1. Critique the inference. Identify confounding, multiplicity, and variance issues and explain how each affects the conclusion.
Read the worked answer
The p = 0.03 is not evidence that supplier A causes higher yield. Supplier is confounded with reactor, season, and temperature: A batches were mostly in Reactor 2 in summer and at a 3 degrees Celsius higher setpoint; B batches were mostly in Reactor 1 in winter. Any of those factors could produce the 3.5-point difference, so the supplier effect is not identifiable from these data. The multiple comparisons create a multiplicity problem: with 11 additional tests at alpha = 0.05, the probability of at least one false positive is about 43%, and finding two is not surprising; those p-values should be adjusted, for example with Bonferroni or false-discovery control, or treated as exploratory. Unequal variances mean the pooled two-sample t-test is inappropriate; Welch's test would be better, but it does not fix confounding. Small samples also make the estimate imprecise and sensitive to one or two batches. The correct conclusion is that the data generate a hypothesis, not a supplier decision.
2. Propose a study design and analysis plan that separates supplier from reactor, season, and temperature.
Read the worked answer
Use a randomized, blocked factorial experiment rather than an observational comparison. Factors: supplier at two levels, reactor at two levels, and temperature setpoint at two or three levels, with the temperature range chosen from operating knowledge. Randomize batch order within blocks to avoid time and season effects, and run both suppliers in both reactors across the same period so season is held roughly constant. Block by reactor and by raw-material lot where necessary. Replicate each condition at least three times, with sample size justified by power for the smallest meaningful yield difference, for example 1.5 points, and the observed standard deviation. Record and control nuisance variables such as humidity, catalyst lot, and operator. Analyze with ANOVA or regression including supplier, reactor, temperature, and the supplier-by-reactor interaction; check residuals and variance homogeneity. If the interaction is significant, the supplier recommendation must be conditional on reactor. Pre-specify the primary analysis and adjust for multiple comparisons. Confirm the result with a short production trial before a plant-wide switch.
3. What interim decision is defensible while the confirmatory study is run?
Read the worked answer
Do not switch suppliers plant-wide next month. The current evidence does not separate supplier from reactor and temperature, so a switch could raise cost or disrupt supply without improving yield. A defensible interim decision is to maintain the current qualified supplier, avoid changing multiple process inputs at once, and run the randomized trial. If the plant needs an immediate action, it can test supplier A only in Reactor 2 under a controlled, randomized comparison at the same temperature setpoint used for supplier B, with a predefined success criterion and a limited duration. That is a trial, not a policy change. The team should also document the cost and lead-time implications of dual sourcing and set a decision gate: switch only if the confirmatory experiment shows a supplier effect that is statistically and practically significant after accounting for reactor and temperature. Communicate that the earlier p-value was exploratory and that the current decision protects the plant from a confounded change.