THE FUTURE OF RESEARCH EVALUATION: A SYNTHESIS OF CURRENT DEBATES AND DEVELOPMENTS

A dynamic and inclusive research system is profoundly important for both science and society to advance fundamental knowledge and understanding and to address increasingly urgent global challenges. But the research system is under pressure due to increasing expectations from multiple actors (including funders, governments and the publishing industry), tensions between dynamics of competition and cooperation, an evolving scholarly communication system, an aggressive – at times – publishing and data analytics industry and limited resources. The research enterprise must manage these demands and tensions while maintaining research quality, upholding research integrity, being inclusive and diverse and safeguarding both basic and applied research.

Over the past decade, these pressures on, and the need for responsiveness of, the science system have been accompanied by more critical reflections on systems of research evaluation and performance measurement. While appropriate, context-sensitive methodologies for assessing research quality and impact are important, debates have intensified about the wide-ranging, complex and ambiguous effects of current evaluation criteria and metrics on the quality and culture of research, the quality of evidence informing policymaking, priorities in research and research funding, individual career trajectories and researchers’ well-being. In some parts of the world, there is a growing recognition that a narrow and simplistic set of evaluative metrics and indicators do not satisfactorily capture the quality, utility, integrity and diversity of research. Routinely used – often journal-based – metrics fail to capture important additional dimensions of high-quality research such as those found in mentorship, data-sharing, engaging with publics, nurturing the next generation of scholars and identifying and giving opportunities to underrepresented groups. In addition to being too narrow in scope, the issue of misapplication of metrics and indicators is also seen to distort incentives for achievement, disadvantage some disciplines (including vital interdisciplinary and transdisciplinary research) and fuel predatory and unethical publication practices.

Campaigns to curb the misapplication of metrics, broaden quality criteria and transform research culture more systematically through manifestos and statements, principles and reforms have set the stage for a global debate on the need to reform research assessment. These voices are now calling for a move from manifestos to action. This is happening against the background of transformational shifts in the ways in which research is being undertaken and communicated. The rise of open research frameworks and of social media, a shift to mission-oriented and transdisciplinary science, the growth in open peer review and the transformative potential of artificial intelligence (AI) and machine learning requires new thinking on how research and researchers are evaluated.

Against this backdrop, the Global Young Academy (GYA), the InterAcademy Partnership (IAP) and the International Science Council (ISC) joined forces to take stock of debates and developments in research evaluation worldwide, drawing on a scoping group of scientists and a series of regional consultations. New approaches are being developed and piloted by higher education institutions and research funders in some parts of the world, and several are included in this paper. In other parts of the world, these debates and actions are nascent or even absent. With research systems evolving at different rates, there is a risk of divergence and fragmentation. Such divergence may compromise the homogeneity needed to enable research collaboration and facilitate researcher mobility across different geographies, sectors and disciplines. However, one size cannot fit all and there is a need for context-sensitive efforts to reform evaluation, recognizing local challenges.

With a focus on public sector research and the evaluation of research and researchers, this discussion paper is global in its perspective, covering an agenda that is typically dominated by developments in Europe and North America: regional perspectives and examples of national  development and institutional reform are highlighted. The global and collective membership of the GYA, IAP and ISC represents a broad cross-section of the research ecosystem whose diverse mandates can facilitate genuine systemic change. This paper endeavours to serve as a stimulus for the GYA, IAP and ISC – as platforms for mutual learning, experimentation and innovation – to work with their members, other science institutions and key constituencies worldwide, to initiate and progress conversations, and mobilize more inclusive and joint action.

Recommendations for the GYA, IAP and ISC and their members (see section 5) are structured around their roles as advocates, exemplars, innovators, funders, publishers, evaluators and collaborators, with indicative timeframes for action. Most immediately, these actions include creating space for sharing lessons and outcomes from relevant initiatives to-date (to build a community of practice); in the medium term, co-convening multistakeholder fora with key constituencies to redesign and implement research evaluation in practicable, context-sensitive and inclusive ways; and, in the longer-term, instigating novel studies that contribute to futures thinking, sensitive to fast-moving developments in AI technologies, peer review methodologies and reform, and communications media.