Interpreting a Group Difference in a Small Training Study
A hospital's education team tests whether a 90-minute simulation improves nurses' recognition of delirium. They recruit 24 nurses and assign 12 to simulation and 12 to a reading packet, using alternating assignment by sign-up order. Recognition is measured by a 10-item vignette quiz immediately after training and again four weeks later. The simulation group scores 8.1 versus 6.4 for the reading group at post-test, with a reported p-value of 0.04; at four weeks the gap narrows to 0.6 points and p = 0.31. Two simulation participants dropped out, and the team lead, who delivered the simulation, also scored the quizzes. The team wants to conclude that simulation is superior and should replace the reading packet.
1. Identify the main threats to the validity of the conclusion that simulation is superior.
Read the worked answer
Several threats stand out. Alternating assignment by sign-up order is not true randomization and may create groups differing in shift, experience, or motivation. The team lead delivered the training and scored the quizzes, introducing expectancy and detection bias; scoring should be blinded. The sample is small, so the p = 0.04 result is fragile and compatible with modest effects. Differential dropout of two simulation participants could bias the comparison. Immediate post-test gains may reflect enthusiasm or test familiarity rather than durable learning, and the four-week result shows the difference largely faded. The claim should be limited to short-term quiz improvement in this sample, with the durability question open. A stronger design would use concealed random assignment, blinded scoring, a pre-specified primary outcome, and adequate power.
2. What should the team report, and what would a reasonable next step be?
Read the worked answer
Report both time points, group means with confidence intervals, the number analyzed, and the dropout details, and state that the immediate difference was not sustained. Note the lack of blinding and the non-random assignment as limitations. A reasonable next step is a larger study with concealed allocation, blinded outcome assessment, a pre-registered primary outcome, and a follow-up long enough to test retention, ideally with a measure of on-the-job behavior rather than quiz scores alone. Meanwhile, the team can keep the reading packet and add simulation as a supplement, since the evidence does not justify replacing a low-cost option on the basis of a small, unblinded, short-lived difference. The decision should weigh cost and feasibility, not just the p-value.
3. A manager says, 'p = 0.04 means there is a 96% chance simulation works.' Evaluate this statement.
Read the worked answer
The statement misreads the p-value. A p-value is the probability of data at least this extreme if the null hypothesis were true; it is not the probability that the hypothesis is true or false. The probability that simulation is better depends on prior evidence, study design, and effect size, and can be estimated with Bayesian reasoning or by considering the fragility of a small trial. With 24 participants, unblinded scoring, and a fading effect, the posterior support for a durable advantage is weaker than the p-value suggests. The manager should be told that the result is compatible with a real but small short-term effect and also with chance. The practical response is to avoid overclaiming, report uncertainty, and gather better evidence before a policy change.