The Pilot That Improved the Average and Barely Touched the Median
An office processes about 64 applications a month with a mean processing time of 41 days, a median of 33 days, and a standard deviation of 16 days. A nine-person pilot team, handling 36 applications over the same period, reports a mean of 34 days, a median of 31 days, and a standard deviation of 15 days. The pilot used a new intake checklist, and staff volunteered to join it. Both samples come from the same four months and the same application categories. The deputy director says the seven-day gain is decisive and wants agency-wide rollout next quarter; the budget office wants one more quarter of evidence before funding 40 positions.
1. Construct an approximate 95% confidence interval for the difference in mean processing time and interpret it in plain language for a non-statistician.
Read the worked answer
Standard errors
- Current process: 16 / sqrt(64) = 16 / 8 = 2.0 days.
- Pilot: 15 / sqrt(36) = 15 / 6 = 2.5 days.
- Standard error of the difference: sqrt(2.0^2 + 2.5^2) = sqrt(4.00 + 6.25) = sqrt(10.25) ≈ 3.20 days.
Interval
- Observed difference: 41 - 34 = 7.0 days.
- Margin: 1.96 x 3.20 ≈ 6.3 days.
- 95% CI: 0.7 to 13.3 days.
Plain-language reading: the data are consistent with a real improvement, but the plausible size runs from under one day to more than thirteen. The interval excludes zero, which is why the result looks encouraging, but its lower bound is so close to zero that a planning decision built on "roughly seven days" is not supported. To plan a 40-position investment, you need an upper bound on the benefit, and this interval does not supply one.
Caveat to state out loud: the interval assumes approximate normality of the sample means. With processing times this skewed (mean 41 against median 33), that assumption is doing real work, and the true uncertainty is likely wider than the interval suggests, not narrower.
2. Reconcile the change in the mean with the change in the median. What does the pattern imply about where the time savings occurred?
Read the worked answer
The two statistics moved very differently: mean fell 7 days (41 to 34), median fell 2 days (33 to 31).
Inference: if typical cases had gotten seven days faster, the median would have moved about as far as the mean. It did not. The mean is sensitive to the tail; the median is not. A seven-day drop in the mean against a two-day drop in the median is the signature of a shrinking upper tail - a subset of unusually slow applications got much faster, while the ordinary case improved only slightly.
Why this matters for the decision: the checklist probably fixed a specific failure mode (incomplete intake packages bouncing back, or rework loops on complex categories). That is a genuine gain, but it is a different claim from "everything is seven days faster," and it changes the rollout case. The correct supporting metrics are the 90th percentile, the count of applications exceeding 60 days, and a breakdown by category - not the mean alone.
Also check: whether the mean improvement is driven by a handful of cases large enough to move a 36-case sample. With n = 36, two or three extreme values can shift the mean by several days.
3. Draft the recommendation you would give the deputy director, addressing design and evidence rather than only statistics.
Read the worked answer
Recommendation: fund a staged, randomized expansion now; decline the agency-wide rollout next quarter.
Design problems with the pilot as run
- Self-selection. Volunteers are, on average, more motivated and more likely to make a new checklist work. The 7-day gap confounds the checklist with the people.
- No concurrent control. A four-month comparison against a prior period cannot separate the checklist from seasonal caseload mix or from other changes made in the same window.
- Unblinded measurement. If the pilot team knew it was being timed, recording behavior may have shifted.
What the next quarter should look like
- Randomly assign incoming applications or intake teams to checklist versus standard practice.
- Pre-register the primary metric as the median and the 90th percentile, with the mean reported as secondary - this matches the goal (fewer slow cases) to the statistic.
- Stratify by application category so the tail effect can be attributed.
- Size the sample to detect a 2-day median shift, not a 7-day mean shift; the current n = 36 cannot support the claim being made.
To the budget office: the pilot is a promising signal, not a demonstrated effect. Release a fraction of the 40 positions tied to the randomized phase, and make the remainder contingent on the pre-registered metrics.