Research Assessment, Metrics, and Responsible Evaluation
What this domain tests
Research Assessment, Metrics and Responsible Evaluation tests whether you can critique indicators and design fairer assessment. Expect DORA, the Leiden Manifesto, CoARA, the Hong Kong Principles, journal impact factor misuse and narrative CVs.
Core rules
- The journal impact factor (JIF) is a journal-level average, not a measure of an article, author or team.
- DORA (2012) calls for assessing research on its own merits rather than venue-based metrics.
- The Leiden Manifesto sets ten principles: quantitative evaluation supports qualitative judgment; measure performance against the mission; protect excellence in locally relevant research; keep data collection open; allow scrutinized indicators.
- CoARA (2022) commits signatories to reform assessment, recognize diverse outputs and use evidence-informed approaches.
- Responsible metrics require field normalization, appropriate databases, transparency and attention to gaming. Altmetrics measure attention, not quality.
- Open access can be an assessment criterion, but it should not become a new simplistic proxy.
Worked example
A hiring panel rejects all candidates without a JIF above 10. Reasoning: this uses a journal-level metric as a candidate-level filter, contrary to DORA and Leiden principles. A better process asks for representative outputs, reads them, considers open-science practices and uses field-normalized indicators only as context. The panel should also disclose criteria and allow narrative evidence.
Traps
Confusing JIF with citation count; treating h-index as portable across databases; assuming altmetric scores equal impact; using open access as a quality proxy; and ignoring disciplinary differences in publication and citation norms.