Outliers, Subgroups and One-Tailed Tests: Supervising a Dissertation Analysis
A doctoral scholar studying stress and academic achievement has data from 68 higher-secondary students. The primary Pearson correlation is r = -0.31, p = .011. A boxplot flags three observations; with them removed, r = -0.18, p = .14. The supervisor says the three are probably data-entry errors and tells the scholar to drop them, but no entry log exists and the questionnaires cannot be traced. The scholar has also run six subgroup analyses by gender and medium of instruction; one subgroup (English-medium boys, n = 11) is significant at p = .04. In the draft, the scholar wants to report one-tailed p-values because 'the direction was obvious', omit the full-sample result as 'noisy', and present the significant subgroup as a finding. A journal reviewer has already asked for the t-test to be replaced by a parametric alternative. The scholar asks you to approve the analysis plan before resubmission.
1. Advise the scholar on how to handle the three flagged observations, distinguishing data cleaning from outcome-driven exclusion.
Read the worked answer
Exclusion is legitimate only when the value is demonstrably wrong: a response outside the scale range, a duplicate ID, an impossible age, or a verified transcription error with a source document. Here none of those conditions is met, so removing the cases because they weaken the correlation is outcome-driven exclusion and would be a questionable research practice.
The defensible approach is a pre-specified primary analysis on all 68 cases, followed by a sensitivity analysis that reports the result with and without the three cases, together with the influence diagnostics that justify discussing them (Cook's distance, leverage, DFBETA). If non-normality or heavy tails drive the instability, report a rank-based correlation or a bootstrap confidence interval as the robustness check rather than switching tests to chase significance. The draft should state the full-sample r = -0.31 with its 95% CI, then note that the association attenuates to -0.18 when three influential cases are excluded, and interpret that attenuation as a substantive finding about heterogeneity, not as an inconvenience. The reviewer's point about distributional assumptions is answered by reporting both Pearson and Spearman estimates and the bootstrap CI.
2. Evaluate the subgroup analysis and the switch to one-tailed tests, and state what the final report must contain.
Read the worked answer
Six unplanned subgroup tests at alpha = .05 produce roughly a one-in-four chance of at least one false positive, and the surviving cell has n = 11, so its estimate will have a very wide confidence interval. The subgroup result can be reported only as exploratory and clearly labelled as such, with the number of tests disclosed and, ideally, a multiplicity correction such as Holm or Benjamini-Hochberg. It cannot be presented as a confirmed moderator, and the full-sample result cannot be dropped because it is inconvenient.
A one-tailed test is acceptable only when the direction was specified before data collection and a negative association is theoretically the only plausible one, which is rarely true for stress and achievement. Switching after seeing the sign halves the p-value without new evidence and should be refused. The report should contain the pre-specified primary hypothesis, all analyses actually run (including non-significant subgroups), effect sizes with confidence intervals, the sensitivity analysis, the multiplicity statement, and a limitations paragraph on sample size. Data and analysis code should be deposited so the supervisor's instruction to delete cases is not the only record of what happened.