The Supervisor's Significance Problem
A doctoral scholar in a state university has completed an online survey of alumni employability. Of 1,500 invitation e-mails sent, 240 usable responses returned - a 16 per cent response rate. The pre-registered hypothesis, that internship duration predicts starting salary, yields r = 0.11 with p = 0.09. The supervisor, facing a submission deadline and an examiner reputed to demand 'significant findings', advises three changes: exclude the 38 respondents who studied part-time, drop two questionnaire items with low factor loadings, and add a new hypothesis about gender differences that 'the data will support'. The scholar has already signed a declaration, filed with the institutional ethics committee, that the analysis plan was fixed before data collection. The committee's rules require prior approval for any change to the approved protocol. The scholar fears that refusal will delay the award by a year and that the supervisor's recommendation letter will be withheld.
1. Identify the methodological and ethical problem in each of the supervisor's three suggestions, and state which you consider most serious and why.
Read the worked answer
All three are post hoc changes made after the data have been seen, but they damage different things.
Excluding the 38 part-time respondents is a selective exclusion with no pre-specified rationale. Dropping a subgroup because it weakens a correlation is data dredging, and it also silently changes the estimand: the reported r no longer describes the population the study claimed to study.
Dropping the two low-loading items is instrument revision in the service of a result. If the scale is genuinely revised, the revised scale must be re-validated in fresh data; otherwise the reported reliability and validity no longer hold.
Adding a gender hypothesis and presenting it as a study hypothesis is HARKing (hypothesising after the results are known). The p-value attached to it does not carry its nominal meaning, because the hypothesis was generated from the same data that test it.
The third is the most serious because it misrepresents the inferential basis itself: the first two leave an audit trail in the sample and the instrument, whereas HARKing converts exploratory noise into a claimed a priori prediction and shades into fabrication. Separately, the 16 per cent response rate is a non-response bias problem that no re-analysis can repair, and every one of the three changes breaches the signed declaration and the committee's protocol rule.
2. The scholar retains the original analysis plan. What must the thesis report about the non-significant result and the response rate, and what one additional analysis would most strengthen the chapter?
Read the worked answer
The non-significant result must be reported as a finding, not buried. Report the effect size with its confidence interval (r = 0.11, 95 per cent CI approximately −0.02 to 0.24), the exact p-value, n = 240, and the pre-registered direction. Avoid the word 'proves'; avoid switching to a one-tailed test after the fact.
The response rate must be stated plainly (240 of 1,500, 16 per cent) with the fieldwork dates and mode, and the report should not generalise to 'alumni' without qualification. Discuss plausible directions of non-response bias: alumni who are employed, contactable and well disposed are more likely to respond, which would probably inflate rather than deflate any salary relationship.
The most useful additional analysis is a non-response / representativeness check: compare respondents with the known alumni population on programme, graduation year and sex, and compare early with late responders on the key variables. If late responders resemble non-respondents, their near-identity to early responders is reassuring evidence against serious bias. A post hoc power calculation is worth a sentence - with n = 240, a true correlation of about 0.18 would be detected at 80 per cent power, so the study was underpowered for small effects - but it must not be used to reinterpret p = 0.09 as 'a trend'.
3. If the supervisor continues to insist, what procedural route should the scholar take, and what would a defensible amended protocol look like?
Read the worked answer
The route is procedural, not confrontational. First, file a written protocol amendment with the institutional ethics committee describing exactly what is proposed (changed sample, changed scale, added hypothesis) and why; the committee, not the supervisor, decides. Second, place the matter before the Doctoral Advisory Committee or Research Degree Committee, which is the body empowered to approve the thesis plan, and record the disagreement in the minutes rather than in corridor conversation. Third, if the supervisor threatens to withhold the recommendation or to block submission, escalate in writing to the Head of Department and, if needed, the university's ombudsperson or the anti-discrimination/ethics machinery.
A defensible amended protocol would: label the gender analysis exploratory in the title of the section and in the abstract; retain the original correlation as the confirmatory test; retain the full sample as primary and treat the part-time exclusion as a clearly flagged sensitivity analysis reported alongside; and if the scale is revised, validate it in a fresh sample rather than re-scoring the same data. The general principle is that post hoc analyses are not forbidden - concealing their post hoc status is.