Why Members Are Not Renewing: A Research Design Under Budget
A professional association with 8,000 members has seen renewal fall from 82% to 71% over two years. The board is convinced the cause is the cancellation of the annual conference, citing a staff survey of 412 members (5% response) in which 61% named networking as a top benefit. But event attendance had already fallen 18% before the conference was cut, and renewal declines are steepest among members with less than three years' tenure. The association has $15,000 and 10 weeks for research. A vendor proposes a 10-question online survey for $8,000. The executive director wants a defensible answer for the board, not a vendor sales pitch. CRM data includes join date, chapter, event history and email engagement; no renewal interviews are currently conducted. You are the research lead.
1. Critique the current evidence and propose a mixed-method research design that fits the budget and timeline.
Read the worked answer
Critique The 412-response survey has a 5% response rate, so it describes respondents, not members. It measures stated importance, not renewal behavior, and it cannot show that cancelling the conference caused the decline. Event attendance fell before the cancellation, and the steepest decline among members under three years points to onboarding or early-value failure. The board is reasoning from a single, weak correlation.
Design within 10 weeks and $15,000
- Use CRM first (week 1): segment renewers, lapsed members and new members by tenure, chapter, event history and email engagement. Compare usage before and after the conference cut.
- Qualitative interviews (weeks 2-4): 15-20 semi-structured interviews, mostly lapsed and new members, plus a few renewers. Ask about the renewal decision in their own words; do not ask 'Did cancelling the conference cause you to leave?'
- Survey (weeks 4-7): invite a random sample of lapsed and current members with a short questionnaire; oversample new members. Include open-ended reasons, benefit usage and renewal intent. Use the vendor only if it provides the sampling frame and raw data.
- Triangulate (weeks 8-10): compare interview themes, survey results and CRM behavior. Report confidence and contradictions, not a single cause.
Allocate roughly $6,000 to incentives and interviewing, $5,000 to survey fieldwork, and the rest to analysis and reporting.
2. Write four survey questions with a rationale for each, and identify the main sampling and analysis issues.
Read the worked answer
- 'In your own words, what was the main reason you decided not to renew?' Open-ended; avoids putting the conference in the respondent's mouth and lets new reasons emerge.
- 'Which of these did you use in the past 12 months: chapter events, online resources, mentoring, advocacy updates, annual conference?' Measures behavior rather than stated importance.
- 'How likely are you to renew in the next 12 months?' Five-point scale; gives a comparable outcome and can be linked to tenure.
- 'What one change would make membership worth renewing for you?' Open-ended; generates testable ideas without implying blame.
Sampling issues: the frame must include lapsed members, not only current members; a random sample is needed, with reminders to reduce nonresponse. A 400-response sample from 8,000 gives roughly a 5% margin of error at 95% confidence, but nonresponse bias may be larger than sampling error. Weight only if reliable population counts exist; otherwise report unweighted results with limitations. For analysis, cross-tabulate by tenure and usage before testing any causal claim.
3. How would you present the findings to the board so that uncertainty is clear but the board can still act?
Read the worked answer
Present a decision memo, not a vendor deck. Start with the decision the board must make: whether to invest in retention, and where. Then show: (1) what the data can support: renewal is declining, and the decline is concentrated among newer members; (2) what it cannot support: that the conference cancellation is the sole or main cause; (3) the qualitative themes and their frequency; (4) the survey results with margins and response rates; (5) a segment map showing where risk is highest.
Use a confidence table: 'strong evidence,' 'suggestive,' 'unknown.' Recommend two low-cost tests: a new-member onboarding sequence and a networking offer for members under three years, each with a 90-day success measure (renewal intent, event attendance, activation). Put a dollar estimate on the retention opportunity but label assumptions.
Close with a measurement plan: quarterly renewal tracking by tenure, a short pulse survey, and interviews with every 20th lapsed member. Tell the board that this will not prove causation, but it will show whether the actions move the right leading indicators. That is more defensible than adopting the conference explanation and spending the budget on a single fix.