Evidence Appraisal and Critical Reading of the Literature
The Evidence Appraisal domain tests whether you can judge a paper's trustworthiness rather than merely summarize it. Expect questions on evidence hierarchies, risk-of-bias instruments, synthesis methods, and the gap between statistical significance and clinical usefulness. You should be able to name the tool that fits the design and the bias domain that most threatens the result.
Choose the appraisal instrument that matches the design
- RCTs: Cochrane RoB 2, covering randomization, deviations from intended interventions, missing data, outcome measurement, and selective reporting.
- Non-randomized studies: ROBINS-I, which appraises confounding and selection of participants explicitly.
- Systematic reviews: AMSTAR-2; reporting quality: PRISMA 2020.
- Diagnostic accuracy: QUADAS-2. Qualitative work: CASP or JBI checklists.
- Guidelines: AGREE II across six domains, including rigour of development and editorial independence.
- Certainty of a body of evidence: GRADE, rating down for risk of bias, inconsistency, indirectness, imprecision, and publication bias; rating up for large effects or dose-response.
Read the numbers, not the abstract
Relative measures exaggerate. Convert to absolute risk reduction and NNT using the study's baseline risk, then ask whether your patient's baseline resembles the trial population. Check confidence intervals for width and for whether they cross the minimally important difference, not just the null.
Worked example
A meta-analysis of eight trials reports a pooled risk ratio of 0.78 (95% CI 0.66-0.92) for a discharge education intervention, with I-squared of 62 percent, and the funnel plot is asymmetric with small positive trials present and small negative trials missing. Reasoning: the pooled estimate is statistically significant, but I-squared above 50 percent signals meaningful inconsistency that a random-effects model does not fix; asymmetry suggests small-study effects or publication bias; AMSTAR-2 would likely rate the review low or critically low if the protocol was unregistered and grey literature unsearched. The honest conclusion is 'probably a modest benefit, certainty low to moderate', not 'proven effective'.
Traps that cost marks
- Equating a low p value with a large or important effect.
- Believing a random-effects model resolves heterogeneity; it widens the interval, it does not explain the variance.
- Accepting surrogate endpoints such as HbA1c or tumour size as patient-important outcomes.
- Missing selective outcome reporting when the registered protocol lists different primary outcomes.
- Treating a guideline recommendation as primary evidence rather than a synthesis of it.