A quasi-experimental M.Phil. study with a weak instrument and a publication dilemma
An M.Phil. scholar at a state university in Assam is studying whether a flipped classroom improves achievement in mathematics among B.Ed. trainees. She has two intact sections. Section A (n = 38) she teaches herself, using pre-class videos and in-class problem solving. Section B (n = 35) is taught by a senior colleague who declined to change his lecture-plus-worksheet method. Both sections take a 30-item achievement test before and after the eight-week intervention. Six trainees dropped out (four from A, two from B). The test was assembled by the scholar from a coaching workbook, and its Cronbach's alpha in the pilot was 0.58. She plans an independent-samples t-test on gain scores and has already told the head of department that the method 'works'. She also asks whether she may publish the same results in a university journal and in a paid conference proceedings under different titles, 'since neither is widely read'.
1. Identify the three most serious validity threats in this design and state the single most defensible analytic strategy available without randomisation, with reasons.
Read the worked answer
Three threats that matter most
- Selection and teacher confound. Sections are intact and taught by different people; Section A also receives the researcher's own enthusiasm. Any post-test difference is jointly attributable to method, teacher and baseline differences, and these cannot be separated.
- Instrumentation. Alpha of 0.58 is far below the usual 0.70 floor for a 30-item achievement test; items drawn from a coaching workbook may be off-syllabus or rote-oriented. Measurement error this large can manufacture or mask an effect.
- Differential attrition and expectancy. Four of 38 leave A and two of 35 leave B, and leavers are rarely random. The researcher has a stated prior ('it works'), which invites scoring bias and demand characteristics.
Most defensible analysis: ANCOVA on post-test scores with pre-test as covariate and section as the grouping factor, reporting adjusted means, 95% confidence intervals and an effect size - not a t-test on gain scores. Reason: gain scores compound measurement error and assume baseline equivalence; when groups differ at baseline they are mathematically biased (the Lord's paradox and regression to the mean). ANCOVA adjusts for the baseline difference. Report the baseline equivalence test anyway, and label the design quasi-experimental: an adjusted association, not proof of causation.
2. She has already told the head of department that the method 'works'. What should she do now, and how does this bear on research integrity?
Read the worked answer
The claim was premature, and it is now a small integrity problem in its own right. Repair it in this order:
- Pre-specify. Write and date the analysis plan - outcomes, covariate, planned comparisons, alpha - and give a copy to the supervisor before examining post-test data. This is what prevents trying five analyses and reporting the one that worked (HARKing).
- Correct the record with the head of department in writing, without drama: the analysis has not been done; the earlier statement was an expectation. A correction now costs far less than a retraction after publication.
- Report what is found, including a null or negative result. A well-conducted study showing no effect is publishable and more useful than a padded positive one.
- Keep the data trail: pilot reliability output, instrument blueprint, attendance records for the eight weeks, reasons for each drop-out.
- Supervisor sign-off before any submission. The supervisor is a co-author or at least a gatekeeper; the scholar should not be the only check on her own conclusions.
3. Advise on the plan to publish the same results twice under different titles.
Read the worked answer
Advise against it. Dual publication of the same results is not made acceptable by the journals being little-read.
- Why it is misconduct rather than a technicality. It inflates the apparent evidence base - a meta-analyst counting both papers gives one study double weight - and it usually breaches the first journal's copyright transfer. Different titles do not create different work; editors run text-similarity checks.
- The narrow exception: transparent overlapping publication, where a full peer-reviewed paper is accompanied by a clearly labelled conference version that cites the full paper and where both editors are informed at submission.
- A legitimate two-paper strategy: paper 1 reports the achievement outcome quantitatively; paper 2 reports genuinely distinct data - engagement logs, or interviews about the video-viewing experience - with explicit cross-citation.
- Risk calibration: proven duplicate publication can lead to retraction and penalties for the author under India's academic-integrity framework. The upside here (a marginally more visible conference paper) is far smaller than the downside.
Recommendation: submit to the peer-reviewed journal, disclose any conference version, and reserve the second paper for a different research question.