'Quantitative literacy' would prevent unsound research policy

Research impact is measured in different ways. However, these indicators are often based on dubious calculations, says Ludo Waltman.

Policymakers try in various ways to evaluate how well researchers and research institutions are performing. How often is a researcher cited by colleagues in the field? How great is his or her impact, as measured in the h-index? And how well is a university performing in comparison with other institutions at home and abroad?

However, these indicators and rankings are often based on dubious calculations, says Ludo Waltman in his inaugural lecture. This Professor of Quantitative Science Studies argues that university rankings combine incompatible indicators. "These rankings measure in part the size of universities and in part their relative performances in relation to their size. These perspectives are then thrown into one pile. Then no one really understands what exactly the rankings are telling us." Other rankings also juggle the figures.

According to Waltman, the danger is that simplistic indicators will have a negative effect on research policy, for instance if researchers or research institutions are assessed on the basis of these indicators. He believes that a higher level of quantitative literacy is needed to bring about change. A better understanding of quantitative analyses—as well as of the conclusions that are reached from these analyses—would make a world of difference.

Waltman believes that quantitative research places too much value on statistical significance, "Which means that research results are presented in an unnatural, binary way, so as significant or not." Furthermore, in these calculations, correlation is often confused with causality. Much could be gained from training that increases levels of quantitative literacy, says Waltman. "The use of data-driven methods in the specific context of policy development doesn't appear to receive the systematic attention that it deserves."

"Wherever, possible, research policy should make use of the findings of quantitative science," says Waltman in his inaugural lecture. "But it is unrealistic to think that we can provide clear, quantitative answers to the many questions that exist. I think that we should move away from the idea that we can answer these kinds of complex question with a simple yes or no."