Data, Indicators and Evidence-Based Decision Making
This domain tests whether you can turn numbers into defensible judgements. EAQE items present a metric, a trend or a dashboard and ask what can and cannot be concluded, which indicator fits the question, and what would make the evidence trustworthy.
Core rules
- Indicator types. Input (staff ratio, facilities), process (assessment turnaround, moderation coverage), output (pass rates, progression), outcome (employability, further study, graduate attributes), and impact. Leading indicators predict; lagging indicators confirm. Choose by the question asked, and never manage outcomes with process metrics alone.
- Data quality dimensions. Validity (measures what it claims), reliability (consistency over time and raters), completeness, accuracy, timeliness, consistency (same definitions across units). A metric failing these is decoration.
- Definitions and metadata. A data dictionary fixes who counts in a cohort, how a pass is defined, whether transfers and withdrawals are included, and when the snapshot is taken. Most "improvements" vanish when definitions are checked.
- Baselines, targets and thresholds. Without a baseline, a target is arbitrary; without a threshold, a target has no consequence. Distinguish aspirational targets from minimum standards.
- Triangulation and disaggregation. Aggregate figures hide variation. Disaggregate by programme, level, mode and equity group; beware small denominators where one student moves several percentage points.
- Causal caution. Correlation, confounding, selection effects and Simpson's paradox can reverse an aggregate trend. Use comparison groups, like-for-like cohorts and, where feasible, before/after designs.
- Benchmarking and rankings. Benchmark against comparable institutions and like-for-like definitions; rankings use different weightings and should not be treated as quality measures.
Worked example
A department reports first-year pass rates rising from 70% to 88% after introducing a new tutorial scheme, and claims the scheme caused the gain. Reasoning: first check the denominator and cohort composition (entry tariff, international share, mode of study, assessment regulations), then whether the pass definition or resit treatment changed, then whether the assessment instrument changed in difficulty. If the cohort's prior attainment also rose, the gain is partly compositional. Defensible conclusion: the improvement is real for the observed cohort but cannot be attributed to tutorials without a comparison group. Recommendation: introduce the scheme in comparable programmes and compare like-for-like, with the same definitions, or track the same cohort's later modules to see if the gain persists.
Traps that cost marks
- Celebrating a rise without checking definitional change or cohort composition.
- Vanity metrics that always look good and drive no decision.
- Over-interpreting small samples and ignoring confidence/uncertainty.
- Gaming: when an indicator becomes a target, behaviour shifts to the indicator.
- Comparing institutions or years that use different definitions.
- Presenting a dashboard with no baseline, target or owner for action.