Designing a Mixed-Methods Study with Limited Resources
A PhD scholar wants to study the impact of a new mentoring program on first-generation students' academic self-efficacy. She has a small budget and limited time (6 months). She proposes a mixed-methods design: a pre-post survey with a comparison group (mentored vs. non-mentored) and semi-structured interviews with 10 mentored students. The survey instrument has 20 items and will be administered online. The comparison group will be students who applied but were not selected due to limited seats. The scholar worries about selection bias and whether the sample size (n=40 mentored, n=40 comparison) is adequate. She also plans to analyze interviews using thematic analysis but has no prior experience.
1. Evaluate the proposed design. What are the main threats to internal validity, and how might they be addressed?
Read the worked answer
Main threats: (1) Selection bias: The comparison group consists of students who applied but were not selected; they may differ from selected students in motivation or other unmeasured ways. This is a nonequivalent control group design. To address, the scholar could collect baseline covariates (e.g., prior GPA, motivation scores) and use statistical adjustment (ANCOVA) or propensity score matching. (2) History/maturation: Over 6 months, other events could affect self-efficacy. Including a comparison group helps control for this. (3) Testing effects: Pre-test may sensitize participants. Could use a Solomon four-group design, but resources are limited. (4) Instrumentation: Online survey may have issues, but likely stable. (5) Attrition: With 6 months, dropout possible; track and report. For interviews, the lack of experience in thematic analysis is a threat to trustworthiness; she should seek training or collaborate with an experienced qualitative researcher. The sample size of 40 per group may be adequate for detecting moderate effects (power analysis needed).
2. How should the scholar integrate the quantitative and qualitative components? Propose a specific mixed-methods design and justify it.
Read the worked answer
Given the timeline and resources, an explanatory sequential design is appropriate: first collect and analyze quantitative data (pre-post surveys), then conduct interviews to explain or elaborate on the quantitative results. For example, if the survey shows a significant increase in self-efficacy for mentored students, interviews can explore how mentoring contributed. Alternatively, a convergent parallel design could be used if time is tight, but explanatory sequential is more feasible for a novice. The scholar should ensure integration by connecting the two phases: select interview participants based on survey responses (e.g., high and low gainers) and use survey findings to inform interview questions. This design is practical, allows for triangulation, and fits the 6-month timeframe if data collection is staggered. The qualitative analysis should be conducted with guidance to ensure rigor.