The Three-Batch Claim: Deciding Whether a Change Actually Improved Output
Your team must tell a program lead by Thursday whether a process change reduced output. The pre-change baseline was 42.0 units per batch, averaged over 30 batches. After the change, three batches measured 47, 51, and 44. The gauge's stated tolerance is plus or minus 3 units at this range, and its calibration certification lapsed five weeks ago. Two operators collected the post-change data; operator B's two readings average 4.5 units above operator A's single reading. The program lead has already told her director that the change "looks positive" and wants your written concurrence before the Thursday review. There is budget for eight more measurements and no budget to re-run the full 30-batch baseline.
1. State precisely what the current data do and do not support, and identify the two largest threats to a valid comparison.
Read the worked answer
The data do not support a conclusion that output changed. Three post-change values of 47, 51, and 44 sit inside a noise envelope that is plausibly 6 to 8 units wide once gauge tolerance (plus or minus 3) and the 4.5-unit operator offset are combined. A single operator-A reading of 44 is essentially indistinguishable from the 42.0 baseline. Nothing here establishes a shift in the process mean.
The two largest threats are measurement system validity and confounding by operator. First, the gauge is out of calibration, so any difference could be instrument drift rather than process change; the tolerance figure is a spec, not a measured current error. Second, the operator effect is perfectly tangled with the batch effect: we cannot tell whether B's high readings reflect B or the batches B measured. A third, smaller threat is that 30 baseline batches and 3 post-change batches give wildly unequal precision. Reporting "looks positive" on this evidence is an overreach.
2. Allocate the eight remaining measurements. Justify the split and the order of operations.
Read the worked answer
Measurement validity comes before more process sampling, because extra process data collected on a suspect gauge only adds uninterpretable numbers.
Suggested allocation: three measurements on a check standard or a retained reference batch, read by both operators, to quantify current gauge bias and the operator offset directly. Then five measurements spread across the new process, with both operators represented so the operator term can be estimated rather than confounded. If budget forces a choice, spend four on the measurement study and four on the process; a process comparison with a known measurement bias can be corrected, one with unknown bias cannot.
The key discipline is that the eight measurements answer two distinct questions. Do not let the program lead's deadline collapse them into one. If the gauge check shows bias larger than roughly 2 units, the process comparison should be reported as inconclusive rather than forced.
3. Draft the substance of the written response to the program lead. What must it contain, and what wording should be avoided?
Read the worked answer
The response should contain four things: an explicit statement that concurrence is withheld, the reason in operational terms, the plan with dates, and the decision the program lead can safely make now.
Substance: "I cannot concur that output increased. Three post-change batches are within the combined uncertainty of the gauge, which is out of calibration, and of an observed operator difference of 4.5 units. Operator and batch effects are currently confounded. We will run a check-standard study and five further process measurements before Thursday, and I will give you a bounded answer then. Until then, please describe the result as 'under evaluation.' If the Thursday review cannot accommodate that, I will attend and state the limitation myself."
Wording to avoid: any adjective that can be quoted out of context, such as "promising," "trending up," or "preliminary positive." Avoid "no change," too, since that is also an unsupported claim. Describe what is unknown, not what you hope.