The Executive Summary
A grant report's executive summary states: '81% of teachers found the literacy workshop useful, demonstrating strong demand for a national roll-out.' You are asked to check the claim before it goes to the funder. The appendix shows: 210 teachers attended; 47 returned the six-month follow-up form; of those, 38 ticked 'somewhat useful' or 'very useful' and 9 ticked 'not useful'. The form also shows that 31 of the 47 respondents had already been using the workshop's core method before they attended. Attendees were recruited by invitation from schools that volunteered. The workshop cost R2,100 per teacher. A separate district survey (n = 1,300, randomly sampled) found that 52% of teachers could not name the method.
1. Rewrite the summary sentence so that it is accurate. State explicitly what the data can and cannot support.
Read the worked answer
Accurate version:
'Of the 47 teachers who returned the six-month follow-up (22% of the 210 who attended), 38 - 81% of respondents - rated the workshop as somewhat or very useful. This indicates positive appraisal among those who responded; it does not establish demand for a national roll-out.'
Can support: a satisfaction finding within the responding group. Cannot support:
- The 81% figure as a share of attendees. 47 / 210 = 22% response; the views of the missing 163 are unknown, and non-responders in follow-up surveys are typically less positive.
- 'Demand' at all - satisfaction with a free, already-attended event is not demand. Demand would require evidence of willingness to pay, which the R2,100 unit cost makes the central question.
- 'National roll-out' from a volunteer-recruited sample of volunteers.
The summary's error is not fabrication; every number is real. It is a scope error - a figure about 47 people is presented as a finding about the teaching workforce.
2. Which single fact in the appendix most undermines the inference of 'strong demand', and why?
Read the worked answer
The most damaging fact is that 31 of the 47 respondents (66%) were already using the core method before attending.
- It changes what the workshop did. For two-thirds of respondents, the workshop cannot have created interest in the method; at most it confirmed it. The 81% is therefore largely a measure of agreement among the already-converted.
- It supplies a competing explanation for the whole result: teachers who had already adopted the method self-selected into a workshop about it, and then rated it favourably. That is closer to a satisfaction survey of existing users than evidence of unmet demand.
The other facts are weaker on their own:
- The 22% response rate threatens generalisability to all 210 attendees, which is a different problem from demand.
- Volunteer recruitment biases the sample but does not by itself explain why respondents were positive.
- The district survey (52% cannot name the method) is the strongest external contradiction, because it suggests low reach - but it measures awareness, not appetite, so it undercuts 'roll-out readiness' more than 'demand'.
The 31/47 figure is the one that breaks the causal story the summary is telling.
3. The funder asks for one change to next year's evaluation so that it could actually support a roll-out decision. What do you propose, and what would it cost you in report length?
Read the worked answer
Proposal: replace the satisfaction survey with a pre/post measure of practice plus a comparison group.
- Record each teacher's method use before the workshop (not retrospectively), then again at six months, alongside a comparison group of teachers who did not attend - from the waiting list, or matched on phase and subject.
- Ask about practice and cost, not only feelings: how often the method is used in a normal week, what it displaced, whether the school would fund it at R2,100.
- Protect the response rate: collect the six-month data during a scheduled contact (a staff meeting or a short call), and report the response rate in the summary itself.
Why this answers the funder's question: a roll-out decision needs to know whether teachers do something differently and whether they would pay, not whether those who replied felt positive.
The cost: the design needs a comparison group, which means more than one page of methods and a results table rather than a single percentage. The summary sentence also becomes longer and less quotable - which is exactly the trade the original report declined to make.