EBP-EAP common mistakes
EBP-EAP · Understand the traps and how to avoid them.
When asked to classify a study by design, the candidate labels any study with a control group as an 'experimental' or 'RCT' design, even when participants were not randomly assigned and were instead studied as intact cohorts or consecutive admissions.
Candidates memorize a hierarchy ladder (case report < cohort < RCT < systematic review) and then classify by surface features such as the presence of a comparison group or the word 'controlled' in the title, rather than by the three defining questions: was there an intervention, was allocation random, and was there a concurrent control. The word 'controlled' in 'controlled clinical trial' is especially misleading because it does not imply randomization.
What to do instead
Before answering, run a three-step checklist out loud: (1) Did the investigators assign the exposure or intervention, or merely observe it? (2) If assigned, was allocation random and concealed? (3) Was there a concurrent comparison group? Only 'assigned + random + concurrent control' is a true experiment. Practice on deliberately ambiguous abstracts where the design label is stripped out, and force yourself to justify the label from methods sentences only, never from the title or the stated conclusion.
The candidate treats 'level of evidence' as a single property of a study and assigns the same level regardless of the clinical question being asked, for example rating a well-conducted cohort study as low-level evidence for a prognosis question or rating a diagnostic accuracy study on the therapy hierarchy.
Most study materials present one hierarchy, usually the therapy hierarchy with systematic reviews of RCTs at the top. Candidates then apply that ladder to every question type, not realizing that prognosis, diagnosis, harm/etiology, and quality improvement each have their own preferred designs and their own appraisal criteria. This produces answers that are internally consistent but wrong for the question asked.
What to do instead
Anchor every level-of-evidence question to the question type first. State the question type explicitly (therapy, diagnosis, prognosis, harm, meaning/qualitative, quality improvement), then recall the design that best answers it: for prognosis, a prospective inception cohort; for diagnosis, a cross-sectional study with an independent blinded reference standard; for harm, cohort or case-control depending on rarity. When a stem mixes designs, answer for the question the clinician actually asked, not for the design that sounds most rigorous.
The candidate writes a PICO question that omits the comparison or the outcome, or substitutes a vague population such as 'patients' or 'elderly people' and an outcome such as 'better outcomes' or 'improved quality of life' without specifying the measure, timeframe, or instrument.
Candidates learn PICO as a memorized acronym and fill in each letter with whatever phrase first comes to mind, treating the exercise as a vocabulary test rather than a search-planning tool. They also fear that adding specificity will 'narrow the search too much', so they keep terms broad, which paradoxically makes the question unsearchable and unanswerable.
What to do instead
Write the question as a single grammatical sentence that must contain all four elements, then stress-test it: could two different clinicians search this and retrieve the same core set of studies? Replace 'patients' with the setting, age range, and condition; replace 'better outcomes' with a named instrument, threshold, and time horizon (for example, a specific scale score at 12 weeks). If the comparison is genuinely 'usual care', say so explicitly rather than leaving it blank. Practice converting vague questions into searchable ones and back again.
The candidate builds a search using only free-text keywords and does not use controlled vocabulary, Boolean operators, truncation, or field limits, and then reports that the literature 'does not exist' or accepts the first few hits as the complete evidence base.
Candidates often search the way they search the web: type a phrase, scan the first page, stop. They are unfamiliar with how subject headings differ from keywords, how synonyms and spelling variants fragment results, and how a single missed synonym can hide an entire body of relevant trials. They also skip the reference lists and related-article links that would reveal what the search missed.
What to do instead
For every search, document four things: the database, the exact string including Boolean logic and truncation, the limits applied, and the date run. Combine at least one controlled-vocabulary term with free-text synonyms joined by OR inside each PICO concept, then join concepts with AND. Supplement the database search with citation chasing (forward and backward) on the key included studies. Before concluding evidence is absent, re-run with synonyms, broader terms, and no date or language limit, and check whether a published systematic review already covers the question.
When appraising a therapy study, the candidate accepts the authors' reported statistical significance as proof of clinical importance, quoting a p value as the main finding and ignoring effect size, confidence interval width, and whether the outcome was patient-important versus a surrogate.
Training and habit reward the p value as a binary verdict. Candidates conflate 'statistically significant' with 'large' and 'clinically worthwhile', and they do not routinely check whether the confidence interval crosses the minimally important difference. Surrogate endpoints such as laboratory values are also easier to report and are accepted without asking whether they translate into outcomes patients feel.
What to do instead
For every result, extract four numbers before judging: the point estimate, its confidence interval, the minimally important difference for that outcome, and the absolute effect in the relevant baseline-risk population. Judge significance and importance separately, and state explicitly whether the confidence interval excludes the minimally important difference. Flag any primary outcome that is a surrogate and downgrade confidence accordingly. Convert relative to absolute risk whenever the stem gives a baseline risk.
The candidate appraises a systematic review by reading its conclusion and abstract only, without checking whether the review had a registered protocol, a reproducible search, duplicate independent screening and extraction, a risk-of-bias assessment, and an appropriate handling of heterogeneity.
Systematic reviews carry an implicit authority, so candidates treat the label 'systematic review and meta-analysis' as a quality guarantee. They also assume that pooling always increases certainty, forgetting that pooling heterogeneous studies can manufacture a precise but meaningless summary estimate. The abstract is short and confident, which suppresses the urge to look further.
What to do instead
Appraise any review against a fixed checklist read from the methods section: protocol or registration, search reproducibility, dual independent reviewers, explicit eligibility criteria, risk-of-bias tool applied per study, and a stated rationale for the pooling model. Then inspect the forest plot for direction and magnitude of individual studies, statistical heterogeneity, and whether the pooled estimate is driven by one large trial. If heterogeneity is substantial and unexplained, report the range of effects rather than the pooled number.
When synthesizing evidence, the candidate counts studies or votes by statistical significance ('five studies positive, two negative, therefore the intervention works') instead of weighing risk of bias, precision, consistency, directness, and publication bias.
Vote counting is intuitive and requires no judgment about study quality, so it survives under time pressure. Candidates also over-weight the number of studies relative to their size and rigor, and they rarely consider that small positive studies are more likely to be published, which biases the tally toward effectiveness.
What to do instead
Synthesize by constructing an evidence table with one row per study and columns for design, sample size, risk of bias, effect estimate with confidence interval, and directness to the PICO population. Then judge the body of evidence on each domain separately and state a direction and strength of recommendation that reflects the weakest link. Actively look for publication bias using funnel-plot asymmetry, trial registries, or a search for unpublished or negative studies before concluding the intervention is effective.
The candidate recommends a practice change based on efficacy evidence alone, without addressing local context, baseline performance, resources, staff and patient preferences, cost, or a plan for measuring whether the change actually took hold.
Candidates treat the recommendation as the endpoint of evidence-based practice rather than the midpoint. They equate 'effective in trials' with 'will work here', overlooking that trial populations, fidelity, and infrastructure differ from the local setting. Implementation and evaluation feel like administrative afterthoughts rather than part of the answer.
What to do instead
Structure every recommendation answer in four moves: the evidence and its strength, the fit to the local population and setting including barriers and facilitators, the implementation strategy with named owners and resources, and the evaluation plan with a measure, baseline, target, and review interval. Explicitly name at least one contextual barrier and how it will be addressed. If the evidence is strong but the local readiness is low, recommend a pilot or staged rollout rather than a full-scale mandate.
In a quality improvement scenario, the candidate proposes a solution and declares success without a measurement plan, or designs an evaluation that measures only at the end (a before-and-after with no run chart, no balancing measures, and no attention to secular trends or regression to the mean).
Candidates confuse quality improvement with project management: identify a problem, implement a fix, done. They are unfamiliar with iterative small-scale testing and with the idea that improvement must be demonstrated over time against a stable baseline. They also omit balancing measures, so they miss unintended harm elsewhere in the system.
What to do instead
For any improvement scenario, specify the family of measures (outcome, process, balancing), the data source and sampling interval, and the display method (run chart or statistical process control chart with a baseline and annotated interventions). Frame the change as a series of small tests of change rather than a single launch. When interpreting a shift, explicitly rule out secular trend, seasonal variation, and regression to the mean before attributing the change to the intervention.
The candidate answers an ethics or interprofessional item by choosing the option that is most clinically efficient or most respectful in tone, rather than identifying the governing principle, the decision-making capacity assessment, or the correct escalation pathway.
Ethics items are written so that one option sounds kind and practical and another sounds procedurally correct but blunt. Candidates select on emotional valence. They also collapse distinct concepts such as capacity, competence, and consent, and they default to 'ask the attending' or 'call the family' without recognizing when patient autonomy, confidentiality, or a formal ethics consultation is the required step.
What to do instead
Classify the item before choosing: is the conflict about autonomy, beneficence, nonmaleficence, justice, or confidentiality, and is the patient's decision-making capacity in question? Apply a fixed sequence for capacity (understand, appreciate, reason, express a choice) and for conflict (direct discussion, then escalation through the defined pathway, then formal consultation). Choose the option that follows the correct process even when a warmer-sounding option exists, and reject options that bypass the patient's own authority when capacity is intact.