Models to predict scientists' future impact often fail

October 30, 2013
Researchers at work.

Models universities partially use to forecast scientists' future contributions are not as reliable as previously thought. In a recent study, Aalto University and IMT Institute for Advanced Studies Lucca researchers demonstrate mathematical analysis of past performance cannot reliably determine the future performance. This means that current models dangerously overestimate the predictability and should not be used for career advancement decision process.

"Based on our results, the predictability of current models for real application in recruitment decisions is questionable. Efforts to model future impact need to be aimed more directly at applications in the career advancement decision process," says Professor Santo Fortunato from Aalto University in Finland.

In recent years it has become more common for universities to use quantitative measures for yard-sticking the productivity and impact of individual researchers to help reduce hiring risks. Models thought to be capable of foreseeing a scientist's future impact by way of his or her future 'h-index' have become a common tool in recruitment decisions as well as other scientific evaluation, advancement, and reward processes.

Scientists concluded that great caution should be taken when attempting to forecast an individual's future based on their 'h-index'. Specifically the authors show that it is easy to grossly overestimate the of cumulative measures over a person's entire career.

The study analyzed 762 scientists from three disciplines: physics, biology, and mathematics. By applying future impact models to these careers, a number of subtle, but critical, flaws in current models were identified. Specifically, the 'h-index' contains false autocorrelation, resulting in a significant overestimation of "predictive power". Moreover, the "predictive power" of these models vary greatly with the career age of scientists, producing least accurate estimates for already risk-burdened early career researchers.

Aalto researchers concluded that care must be taken to select the correct measures and methods to evaluate scientific candidates in the future. Increased attention should be paid to the potential shortfalls of quantitative methods when applied to the decision-making process.

The results were published in the Scientific Reports, a research publication from the publishers of Nature.

Explore further: New formula predicts if scientists will be stars

More information: Penner, O. et al. On the Predictability of Future Impact in Science, Scientific Reports, 2013. e-print; arXiv:1306.0114

Related Stories

New formula predicts if scientists will be stars

September 12, 2012

A medical school committee is weighing whether to hire a promising young neuroscientist. Will she have a brilliant future as a researcher, publish in top journals and nab abundant research funds?

Novel methods to support decision making processes

January 24, 2013

Mathematical analysis based on numbers and numerical estimates is widely used in decision making everywhere from public administration and business to environmental conservation. Methods for decision analysis evaluate different ...

New IT tool predicts book sales prior to publication

June 25, 2013

In Granada, researchers have developed a new IT system that predicts the volume of sales a book will have if it is finally published. This information will be of great value to publishers, who could use it to decide on the ...

Researchers use science to predict success

October 7, 2013

We all want to know the secret to success and physicists are no different. Like the rest of the academic community, physicists rely on various quantitative factors to determine whether a researcher will enjoy long-term success. ...

Recommended for you

The couple who Facebooks together, stays together

July 27, 2015

Becoming "Facebook official" is a milestone in modern romance, and new research suggests that activities on the popular social networking site are connected to whether those relationships last.

Oldest known Koran text fragments discovered

July 23, 2015

Two pages of text written on parchment that are believed to be sections of the Koran (Chapters 18 and 20) have been discovered by a PhD student in a British university library and are believed to be the oldest ever found. ...

First evidence of farming in Mideast

July 22, 2015

Until now, researchers believed farming was "invented" some 12,000 years ago in the Cradle of Civilization—Iraq, the Levant, parts of Turkey and Iran—an area that was home to some of the earliest known human civilizations. ...

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.