Evaluating a School Mentoring Program With Weak Data
A community agency runs a mentoring program pairing at-risk middle schoolers with adult volunteers for 20 weekly sessions. The program director asks you to evaluate it to support a grant renewal. Available data: attendance logs, a satisfaction survey completed by 61 of 140 participants, and mentor-reported impressions. Last year the agency reported that 82% of mentees 'improved' based on mentor ratings, but the funder now wants evidence of outcomes. There is no comparison group, no pre-test, and the survey was collected only from students who attended at least 12 sessions. The grant renewal is due in 10 weeks, and the agency has no research staff.
1. What are the key threats to validity in the existing data, and what can honestly be claimed from it?
Read the worked answer
The main threats are selection and attrition bias. The 61 respondents are those who stayed at least 12 sessions, so they are likely the more engaged and better-functioning youth; the non-responders and dropouts, who may be the most at-risk, are missing. Mentor ratings are not independent: mentors have a stake in the program's success and may rate improvement to justify their effort, which is expectancy and rater bias. There is no pre-test, so no within-person change can be estimated, and no comparison group, so maturation, history, and regression to the mean are uncontrolled. The 82% figure is a proportion of a biased subsample, not an outcome rate.
What can honestly be claimed: participation and retention figures, satisfaction among attendees, and a description of who was served and who dropped out. The agency should not claim that mentees improved relative to non-participants. Framing the 82% as 'mentor-reported perceived improvement among attendees' is the most that the data support, and even that should be qualified.
2. Given 10 weeks and no research staff, what evaluation design and measures would you recommend?
Read the worked answer
I would design something feasible rather than ideal. The core is adding a pre-post measure with a comparison group that already exists. For the comparison, I would use a waitlist or matched non-participants, or a cohort comparison with the previous year's intake, acknowledging the limits of each.
Measures should be brief, validated, and low-burden: a school-connectedness or student engagement scale, attendance and grades from school records (which avoid self-report bias), and a standardized mentor-mentee relationship quality measure. I would collect baseline at intake and post at the end of the 20 weeks, and track attendance and dropout reasons for everyone, not only completers.
The 10-week timeline means I would not try to run a full cycle. I would conduct an implementation and process evaluation now (fidelity, dosage, retention, who drops out and why), analyze existing administrative data properly with denominators, and build the pre-post design into the next cohort so the renewal application describes a credible plan with a baseline. That is more persuasive to funders than inflated claims.
3. The director says, 'Just write it up so it looks good; everyone does this.' How do you respond?
Read the worked answer
I would name the ethical issue directly but collaboratively. Misrepresenting evaluation results to a funder is not a reporting style; it is deception with consequences for the agency, the youth, and future funding. I would explain that funders increasingly scrutinize claims, that a discovered overstatement risks the renewal and the agency's reputation, and that a modest, well-framed honest report with a credible improvement plan is more competitive than an inflated one.
I would offer a constructive path: reframe the report around what the data do show (reach, retention, satisfaction, implementation), be explicit about limitations, and present the new design as the response to the funder's request for outcome evidence. I would show the director draft language that is honest and still persuasive.
If pressed to misstate findings, I would decline to attach my name, document the request, and consult my supervisor or professional association. I would also check the grant's reporting requirements and any institutional review obligations, since the evaluation may involve human subjects.