A Survey With Too Many Tests: Defending Digital-Learning Readiness
A doctoral scholar submits a draft paper on “digital learning readiness” based on a survey of final-year students. She distributed an online questionnaire through WhatsApp groups of 12 colleges; 298 of 480 eligible students responded. The main scale has six five-point Likert items; Cronbach’s alpha is 0.61. She sums the items into a score and compares male and female students using an independent-samples t-test. She also tests 12 correlations between readiness and variables such as income, device type, hours of study, and parental education. One correlation, with device type, yields p = 0.048; she reports it as “significant” and builds a policy recommendation around it. Income is missing for 22% of respondents, and those with missing income have lower readiness scores. The supervisor asks her to defend the design in a pre-submission seminar and to state what she would change before resubmission.
1. Identify the most serious sampling and measurement problems in the design and explain how each affects the paper’s claims.
Read the worked answer
The sampling frame is not probability-based: the questionnaire was circulated through WhatsApp groups, so respondents self-select and the 298 respondents cannot be treated as a representative sample of the 480 eligible students, let alone of final-year students generally. The 62% response rate is acceptable only if non-response is random, which is not established. Missing income is worse: 22% are missing, and those missing have lower readiness scores, which suggests missingness is related to the outcome. Complete-case analysis would therefore bias the income-related results.
Measurement is also weak. Cronbach’s alpha of 0.61 is below the conventional 0.70 threshold for research use, so the six-item scale needs revision or further validation. Likert items are ordinal; summing them and using a t-test may be defensible if the summed scale behaves approximately interval and the distribution is acceptable, but the paper must justify this rather than assume it. Device type is categorical; correlating it with readiness requires careful coding and is not automatically meaningful as a Pearson correlation.
2. Interpret the p = 0.048 result in light of the 12 tests and state the appropriate conclusion.
Read the worked answer
A p-value of 0.048 means that, if the null hypothesis of no association were true, a result at least this extreme would occur about 4.8% of the time in a single pre-specified test. It does not mean there is a 95% chance the association is real. More importantly, 12 tests were run. The familywise error rate is approximately 1 − 0.95^12 ≈ 0.46, so there is a substantial chance of at least one false positive. A Bonferroni threshold is 0.05/12 ≈ 0.0042. At that threshold, 0.048 is not significant.
The appropriate conclusion is that the device-type correlation is exploratory and not robust to multiplicity correction. The paper should report all 12 tests, not only the significant one, and should give effect sizes and confidence intervals. If the analysis was not pre-specified, it should be labelled exploratory rather than used as the sole basis for a policy recommendation.
3. Recommend a revised analysis and reporting plan before resubmission.
Read the worked answer
First, reframe the study as a convenience-sample exploratory study or obtain a probability sample with follow-up of non-respondents. Second, revise or validate the readiness scale; report item-total statistics and consider dropping weak items, then re-estimate alpha. Third, pre-specify a small number of primary hypotheses and treat the rest as secondary or exploratory, with multiplicity control or clear labelling. Fourth, handle missing income through multiple imputation if missing-at-random is plausible, or through sensitivity analyses and explicit reporting of the missingness pattern; do not silently drop cases. Fifth, use methods appropriate to the measurement level, or justify summated-scale t-tests with distributional checks. Finally, report effect sizes, confidence intervals, and limitations, and avoid policy conclusions that outrun the design.