Models to predict scientists' future impact often fail

Oct 30, 2013
Researchers at work.

Models universities partially use to forecast scientists' future contributions are not as reliable as previously thought. In a recent study, Aalto University and IMT Institute for Advanced Studies Lucca researchers demonstrate mathematical analysis of past performance cannot reliably determine the future performance. This means that current models dangerously overestimate the predictability and should not be used for career advancement decision process.

"Based on our results, the predictability of current models for real application in recruitment decisions is questionable. Efforts to model future impact need to be aimed more directly at applications in the career advancement decision process," says Professor Santo Fortunato from Aalto University in Finland.

In recent years it has become more common for universities to use quantitative measures for yard-sticking the productivity and impact of individual researchers to help reduce hiring risks. Models thought to be capable of foreseeing a scientist's future impact by way of his or her future 'h-index' have become a common tool in recruitment decisions as well as other scientific evaluation, advancement, and reward processes.

Scientists concluded that great caution should be taken when attempting to forecast an individual's future based on their 'h-index'. Specifically the authors show that it is easy to grossly overestimate the of cumulative measures over a person's entire career.

The study analyzed 762 scientists from three disciplines: physics, biology, and mathematics. By applying future impact models to these careers, a number of subtle, but critical, flaws in current models were identified. Specifically, the 'h-index' contains false autocorrelation, resulting in a significant overestimation of "predictive power". Moreover, the "predictive power" of these models vary greatly with the career age of scientists, producing least accurate estimates for already risk-burdened early career researchers.

Aalto researchers concluded that care must be taken to select the correct measures and methods to evaluate scientific candidates in the future. Increased attention should be paid to the potential shortfalls of quantitative methods when applied to the decision-making process.

The results were published in the Scientific Reports, a research publication from the publishers of Nature.

Explore further: Less privileged kids shine at university, according to study

More information: Penner, O. et al. On the Predictability of Future Impact in Science, Scientific Reports, 2013. e-print; arXiv:1306.0114

add to favorites email to friend print save as pdf

Related Stories

Researchers use science to predict success

Oct 07, 2013

We all want to know the secret to success and physicists are no different. Like the rest of the academic community, physicists rely on various quantitative factors to determine whether a researcher will enjoy ...

Novel methods to support decision making processes

Jan 24, 2013

Mathematical analysis based on numbers and numerical estimates is widely used in decision making everywhere from public administration and business to environmental conservation. Methods for decision analysis evaluate different ...

New IT tool predicts book sales prior to publication

Jun 25, 2013

In Granada, researchers have developed a new IT system that predicts the volume of sales a book will have if it is finally published. This information will be of great value to publishers, who could use it to decide on the ...

New formula predicts if scientists will be stars

Sep 12, 2012

A medical school committee is weighing whether to hire a promising young neuroscientist. Will she have a brilliant future as a researcher, publish in top journals and nab abundant research funds?

Recommended for you

Why are UK teenagers skipping school?

Dec 18, 2014

Analysis of the results of a large-scale survey reveals the extent of truancy in English secondary schools and sheds light on the mental health of the country's teens.

Fewer lectures, more group work

Dec 18, 2014

Professor Cees van der Vleuten from Maastricht University is a Visiting Professor at Wits University who believes that learning should be student centred.

How to teach all students to think critically

Dec 18, 2014

All first year students at the University of Technology Sydney could soon be required to take a compulsory maths course in an attempt to give them some numerical thinking skills. ...

User comments : 0

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.