Four Q1 Papers or a Research Narrative?
A university senate is revising its promotion criteria. The draft requires candidates to list at least four publications in Q1 journals, defined by Scopus CiteScore quartile, within the past five years; a departmental committee will also compute a field-normalised citation percentile; candidates may attach a short research narrative. Faculty responses are divided. Humanities and law scholars note that monographs and national-language journals are largely outside Scopus. Early-career researchers point to parental leave and part-time contracts. The library reports that 38% of the journals in which the university's staff publish are not indexed in Scopus, and that quartile assignments shift year to year. National policy endorses DORA and the Coalition for Advancing Research Assessment. A dean wants a benchmark against peer universities using the same list.
1. Critique the draft criteria and propose a replacement clause for the promotion policy.
Read the worked answer
What is wrong with the draft.
- Level error. A journal quartile is a property of the venue, not the paper. A strong article in a Q2 journal and a weak one in a Q1 journal are treated identically.
- Instability. Quartiles are recalculated annually and depend on the journal set included; a paper's classification can change after publication.
- Coverage bias. Monographs, book chapters, law reviews and national-language journals are under-represented in Scopus, so the rule systematically disadvantages humanities, law and applied fields. The library's 38% figure shows this is not a marginal effect.
- Threshold gaming and perverse incentives. "At least four" invites salami-slicing and venue shopping, and penalises those with career breaks or fractional contracts.
- Metric misuse. A field-normalised citation percentile is more defensible than a journal quartile but still weak for books, non-English outputs and emerging fields.
Replacement clause (draft): "Candidates will present a research narrative describing their most significant contributions and their trajectory, supported by up to five outputs of any type, which may include monographs, datasets, software, policy reports, exhibitions or peer-reviewed articles. Reviewers will assess significance, rigour, originality and contribution to the field, using external peer review where appropriate. No journal-level metric may be used as an eligibility threshold or a ranking device. Career breaks and fractional appointments will be considered."
Add: evaluator training, a statement that metrics may inform but not substitute for judgement, and an appeals route. Align the wording explicitly with DORA and CoARA commitments the university has signed.
2. How should the senate respond to the dean's request to benchmark against peer universities using the same list?
Read the worked answer
Separate two questions the dean is conflating: institutional benchmarking, which is legitimate, and using a publication list to rank individual staff, which is not.
What can be done. Institutional-level comparison using transparent, contextual indicators: research income by field, collaboration patterns, open access uptake, doctoral completion, and - with caveats - field-normalised citation indicators at the aggregate level. Publish the methodology, the data source, the year and the limitations. Where peer institutions differ in disciplinary mix, normalise or compare like with like.
What should be refused. Reusing the promotion list as a benchmarking instrument. It would (a) create pressure to optimise for the indicator, (b) convert a developmental criterion into a target, and (c) reward the same coverage bias the library identified. Once a metric becomes a target, departments will recruit and publish to it.
Practical answer to the dean. Offer a benchmarking report built on aggregate indicators plus a qualitative comparison of peer institutions' promotion policies, which is often what is actually wanted: reassurance that the university is not out of step. Note that several peers are themselves moving away from journal-based thresholds. Ask the dean to state the decisions the benchmark will inform; if the answer is resource allocation, the conversation should be about research strategy, not journal lists.
Keep the distinction in writing so the agreed benchmarking dataset cannot later be repurposed as an individual evaluation tool.
3. The library offers a bibliometric service. What should it agree to provide to promotion committees, and what should it decline?
Read the worked answer
Provide:
- Contextualised data for a candidate's outputs: citation counts with field and age normalisation, clearly labelled as indicative.
- Verification of the publication record (correct DOIs, no duplicates, correct affiliations), which is often the real need behind a metrics request.
- Explanatory briefings for committee members on what each indicator does and does not measure, including gaming and coverage limits.
- Aggregate, institutional-level analyses for strategy and benchmarking.
- Support for narrative CVs: helping candidates describe contributions without smuggling in a ranked list.
Decline:
- Producing a single score or rank ordering of candidates.
- Applying journal-level indicators as a proxy for article quality or as an eligibility filter.
- Running analyses of individuals without their knowledge, or supplying numbers to committees that have not been trained in their interpretation.
- Comparing candidates across fields using unnormalised raw counts.
Governance. Publish the service's terms of use, require a stated purpose for each request, and log requests so patterns of misuse become visible. Have the service report to the research committee rather than to individual managers. This converts an ad hoc service into an institutional safeguard and gives the library a defensible basis for saying no.