Seven of Twelve: Calling Significance on a Triangle Test
A roastery's QC lead runs a triangulation to decide whether two drum profiles, A and B, should stay as separate products. Twelve trained staff each receive one triangle: two cups of A, one of B, coded, randomized, served at 60 °C, instruction 'pick the odd cup'. Seven of the twelve choose correctly. The QC lead reports 'a significant difference at p < 0.05; the profiles are distinguishable, keep both'. Two weeks later it emerges that this was the fourth triangle test between the same two profiles: the earlier runs scored 5/12, 4/12 and 6/12 and were not reported, the same three-digit codes were reused in all four sessions, and no warm-up sample was served before any of them.
1. Is 7/12 significant? Show the arithmetic behind your answer.
Read the worked answer
Set up the null model. In a triangle test the taster has three cups, two identical, so guessing gives 1/3. Each taster contributes one independent trial: n = 12, p = 1/3 under the null.
Tail probability for 7 or more correct:
- P(X = 7) = C(12,7) × (1/3)^7 × (2/3)^5 ≈ 0.0477
- P(X = 8) ≈ 0.0149, P(X = 9) ≈ 0.0033, P(X = 10) ≈ 0.0005, P(X = 11–12) ≈ 0.00002
- P(X ≥ 7) ≈ 0.066
Interpretation. 6.6% is larger than the 5% the QC lead claims, so the null is not rejected. The minimum for significance at n = 12 is 8 of 12 (P ≈ 0.019).
Note a subtlety worth stating aloud: the test is legitimately one-tailed. Nobody runs a triangle to detect a difference in the wrong direction, so the criterion is an excess of correct choices, not a two-sided deviation.
So the claim collapses on the numbers alone - before any question about the design. Seven of twelve is exactly the result you would expect from twelve people who can taste nothing at all about a third of the time.
2. Beyond the arithmetic, what is wrong with how this was run and reported, and what is the corrected conclusion?
Read the worked answer
Selective reporting / optional stopping. Four tests were run, three were withheld, and the one closest to significance was published. If each test has a 5% chance of a false positive, the family-wise rate across four looks is roughly 1 − 0.95⁴ ≈ 18.5%. The reported p-value does not describe the decision that was actually made, which was 'run it again until it looks significant'.
Codes reused across sessions. Tasters who see the same codes repeatedly can carry a code-to-sample association between sessions, which is a memory task, not a sensory one. Every session needs fresh random codes.
No warm-up. The first sample a taster meets sets an anchor; without a warm-up triangle, position and order effects are uncontrolled.
Corrected interpretation. Pooling everything the panel actually produced: 7 + 5 + 4 + 6 = 22 correct out of 48. Under p = 1/3 the expected count is 16 with SD ≈ 3.3; 22 lands right around the 5% line - borderline at best. That is not evidence of a commercially meaningful difference between the profiles; it is evidence that the test as designed cannot detect anything short of a very large difference.
Report should read: 'No reliable difference demonstrated. Prior runs were not reported; pooled data are borderline and do not support a product decision on their own.'
3. You are asked to re-run the test because the QC lead is certain a real difference exists. What would make a re-run informative?
Read the worked answer
Assume the true difference is small and buy statistical power rather than hoping for a lucky panel:
- Pre-register the plan: number of sessions, number of tasters, the stopping rule, and the analysis - all fixed before tasting. One look at the data, one decision.
- Size it properly. Twelve tasters cannot resolve a small difference. At p = 1/3, a 24-trial design needs about 13 of 24 correct (12 of 24 gives P ≈ 0.068, still short). More tasters × more trials is the only lever that moves power.
- Fresh random codes every session, randomized balance so each taster's odd cup is A half the time and B half the time.
- Standardize delivery: same brew ratio, same temperature at the moment of tasting (60 °C is fine but it must be enforced, not aspirational), same cup ware, same lighting, a warm-up triangle that is not scored.
- Blind the server, and keep the two people who built the profiles out of the room.
- Decide what counts as actionable. Statistical significance is not the business question. Ask instead: if the difference is real, can a cupper describe it, does it change the cupping score, and does it justify carrying two SKUs? A statistically detectable difference nobody can name is a reason to merge the profiles, not to keep both.