There are many ways for research to be influential, not just citations

Recently, Paul Boon contacted me regarding an email he had received from a US colleague concerning a paper he published recently in Pacific Conservation Biology on the mental health of conservation biologists (Boon 2022). Paul’s correspondent thanked him for the paper, explaining that it was the basis of useful discussions at a lab meeting. It is great to see work used in such ways, although sadly such valuable contributions are invisible in a research evaluation environment geared to metrics. It set me thinking about why metricated evaluations are attractive, what can go wrong with them, and how feedback such as Paul received might be documented and fed into research evaluations, especially those used for appointments and career progression.

Metricated evaluations, such as citation counts or the Journal Impact Factor (JIF), are one means to evaluate the quality of research or the merits of researchers. Superficially, they have several seductive properties. According to their advocates, citations show networks of significant/influential thinkers and ideas (Davies and Calma 2019), ‘the number of citations reflects an article’s influence and therefore quality’ (Wade 1975, p. 429), citations are objective measures (Bavelas 1978), they ‘… reflect the dynamic interplay of interests of both scholars and their institutions’ (van Wesel 2016, p. 199) and allow researchers to screen papers for importance before reading them (Sud and Thelwall 2014). With regard to journal ranking, the JIF is held to quantify research trends and provide a valuable means of prioritising funds for library subscriptions (Jacso 2012), as well as having a simple, clear definition (Bollen et al. 2009). In sum, these measures answer regulatory demand for ensuring that public research money is spent responsibly, often more quickly and cost-effectively than could be achieved by peer review (Hodge and Lacasse 2011; Buela-Casal and Zych 2012), while providing the seeming objectivity and reliable quantification of a numerical system (Adler et al. 2008).

Nevertheless, even advocates acknowledge that assessments based on metrics are highly sensitive to the methods used in individual studies: ‘An analyst (sic) of the results should keep in mind that the identification of landmark papers depends on the used methods and data. Small differences in methods and or data may lead to other results’ (Thor et al. 2021, p. 419). As an example, Miccoli and Rumiati (2019) used metrics to claim a significant increase in Italian scientific productivity, whereas Baccini et al. (2019a, 2019b) and Abramo et al. (2021) argued that the results reflected manipulation of the metrics through self-citation or citation clubs. In another case, Butler (2003, 2017) and Martin (2017) claimed that output-based research funding in Australia led to more papers of lower quality, whereas van den Besselaar et al. (2017) disagreed, claiming that the data showed increases in both quantity and quality of outputs.

‘When a measure becomes a target, it ceases to be a good measure’ (Goodhart’s Law, as quoted by Crawford 2017). Whether or not this statement was actually made by Charles Goodhart (a British economist active in the mid 20th century), it goes to the core of the problem: academics anxious to maximise their prospects may concentrate on good scores, not on good research (Fire and Guestrin 2019). This shift in focus could change what topics they research, what they publish and where they publish. The main drivers for choosing research topics could be restricted to subjects/areas likely to garner citations or appeal to perceived top quality journals, with some authors already offering guides to publishing highly cited papers (Pyke 2013, 2014). Thus, according to some critics: ‘The race for higher rankings has resulted in blind pursuit of scientific publications, with the sole focus being the publication itself, with little or no significance given to the scientific part of it’ (Lamba 2021, p. 176), with ‘The consequence... that many papers have become competitive tokens for insertion into grant-dispensing gambling machines rather than bricks in the edifice of science’ (Lawrence and Locke 1997, p. 758).

Instead of positive outcomes in research outputs, the consequences can be a narrowing of research scope (Martin 2011, 2012), a preference for productive ‘safe research’ over genuine innovation (Martin 2000; Charlton and Andras 2008), an obsession with personal metrics with characteristics of a psychological disorder (Buela-Casal 2014), encouragement of aggressive, acquisitive and exploitative behaviour (Lawrence 2002; Fong and Wilhite 2017) and possibly an incentive to fraud (Chevassus-au-Louis 2019). Anyone who doubts these points need look no further than the attempted manipulation of statistics in journal evaluation (Falagas and Alexiou 2008), Fire and Guestrin’s (2019) critique of publication metrics as an example of Goodhart’s Law in action, or what Biagioli (2016, p. 201) called ‘metrics-enabled fraud’ or ‘post-production misconduct’ in which authors ‘use fraudulent means to secure their publication, enhance their impact and inflate … importance.’

There may also be personal costs for academics struggling to meet performance guidelines based on citation metrics (Parr 2014). Some critics have suggested that metrication is encouraging overproduction of papers at the expense of genuine quality and that there should be a move away from simple productivity metrics to close evaluation of subsets of total output (e.g. Pacchioni 2018; Chevassus-au-Louis 2019). Alan Finkel, Australia’s Chief Scientist in 2019, exhorted us ‘… to heed growing calls to abandon paper counting and similar metrics for evaluating researchers’ (Finkel 2019). The challenge is to create a framework that encourages genuine quality and productivity while minimising risks of misrepresentation (Fire and Guestrin 2019). We have good international guidelines in the Leiden Manifesto (Hicks et al. 2015) and the 2012 San Francisco Declaration on Research Assessment (DORA) (https://sfdora.org), which at least some Australian universities are embracing (https://www.unimelb.edu.au/newsroom/news/2020/july/university-signs-up-to-international-agreement-for-best-practice-in-research-assessment).

Irrespective of criticism and debate surrounding their use, traditional metrics such as the JIF remain the dominant force/criterion in academic review, promotion and tenure in at least North America (McKiernan et al. 2019) and possibly the UK (Else 2021), although very different assessments of academic work are emerging outside the English-speaking world. For example, the ‘Room for everyone’s talent’ initiative of several Dutch univerisities (https://www.universiteitenvannederland.nl/recognitionandrewards/wp-content/uploads/2019/11/Position-paper-Room-for-everyone’s-talent.pdf) ‘… calls for a system of recognition and rewards of academics and research that:

1. Enables the diversification and vitalisation of career paths, thereby promoting excellence in each of the key areas [education, research, impact, leadership and (for University Medical Centres) patient care];

2. Acknowledges the independence and the individual qualities and ambitions of academics as well as recognising team performances;

3. Emphasises quality of work over quantitative results (such as number of publications);

4. Encourages all aspects of open science; and

5. Encourages high quality academic leadership.’

Under such a system, measures of research success other than highly-cited papers in traditional journals could be advanced.

This brings us back to the comment Paul received on his paper, which shows that it set people reflecting even though they may not necessarily write a paper and cite Paul’s work. It is a comment worth making in a detailed assessment of a subset of papers submitted for review or assessment, although it would be missed in a larger, metricated analysis. Perhaps it is time to consider such alternative approaches more seriously.

The author is the Editor-in-Chief of Pacific Conservation Biology.

This research did not receive any specific funding.