Evaluating a Pre-Post Wellbeing Workshop: Attrition, History and Effect Size
A consultancy evaluates a two-day 'resilience at work' workshop. Seventy-two employees of a call centre are enrolled; 60 complete both the pre-test and the post-test four weeks later. The pre-test mean on a 100-point resilience scale is 62 and the post-test mean is 71. The standard deviation of the difference scores is 12. A paired t-test gives t = 5.8, p < 0.001, and the report concludes the workshop caused a 'large, 9-point improvement' and recommends organisation-wide rollout. The 12 employees who dropped out had a mean pre-test score of 48, and most cited night-shift scheduling conflicts. Attendance was voluntary and attendees received a ₹2,000 bonus. The scale has a reported test–retest reliability of 0.85 over four weeks, and the call centre had simultaneously introduced a new incentive scheme that altered shift allocation. No control group was used.
1. Assess the causal claim. Identify the specific threats in this design and quantify the effect size properly.
Read the worked answer
Causation is not supported. One group measured twice has no counterfactual: history, maturation, testing and instrumentation all remain open.
- History: the new incentive scheme changed shift allocation in the same four weeks, so shift timing and fatigue confound the scores.
- Attrition: 12 of 72 (17%) dropped out and their mean pre-test score of 48 is 14 points below completers. Removing them mechanically lifts the post mean. If those 12 are assumed to score as before, the 72-person post mean is (60 × 71 + 12 × 48)/72 = 67.2 - a gain of 5.2 points, not 9.
- Regression to the mean: modest here, since the group was not selected on low scores, but the volunteer-plus-bonus recruitment invites demand effects and expectancy.
- Testing: with test–retest reliability of 0.85, some retest familiarity is folded into the gain.
Effect size: d_z = 9/12 = 0.75 - respectable, not 'large' by conventional thresholds, and nine raw points on an unreferenced scale is meaningless without the scale's minimal important difference.
2. Redesign the evaluation to answer the causal question, and state how the new design would change the conclusion if the effect were illusory.
Read the worked answer
Design: randomise to workshop-now versus a waitlist control that receives the workshop after the post-test. Both arms are measured on identical schedules, so history and testing affect both. Pre-register the primary outcome and the analysis plan.
Attrition: analyse all 72 on an intention-to-treat basis, publish the attrition rate, and follow up dropouts - night-shift conflicts are a scheduling problem, so run sessions inside shift time rather than blaming the participants.
Measurement: collect outcomes from anonymised self-report or administrative records so the ₹2,000 bonus and the consultancy's commercial interest do not leak into the data.
Statistics: report the between-group difference in change with a confidence interval, and an effect size using the control-group or pooled SD - not the raw gain. Power the study for the smallest effect worth acting on.
How the conclusion changes: if the gain survives a waitlist comparison and shift control, it is real and rollout is justified. If it disappears once the 12 dropouts are counted and the control group exists, the 9 points were scheduling, selection and expectancy - and organisation-wide rollout would waste money on a programme that looks good only in completers.