Models to predict scientists' future impact often fail

Oct 30, 2013
Researchers at work.

Models universities partially use to forecast scientists' future contributions are not as reliable as previously thought. In a recent study, Aalto University and IMT Institute for Advanced Studies Lucca researchers demonstrate mathematical analysis of past performance cannot reliably determine the future performance. This means that current models dangerously overestimate the predictability and should not be used for career advancement decision process.

"Based on our results, the predictability of current models for real application in recruitment decisions is questionable. Efforts to model future impact need to be aimed more directly at applications in the career advancement decision process," says Professor Santo Fortunato from Aalto University in Finland.

In recent years it has become more common for universities to use quantitative measures for yard-sticking the productivity and impact of individual researchers to help reduce hiring risks. Models thought to be capable of foreseeing a scientist's future impact by way of his or her future 'h-index' have become a common tool in recruitment decisions as well as other scientific evaluation, advancement, and reward processes.

Scientists concluded that great caution should be taken when attempting to forecast an individual's future based on their 'h-index'. Specifically the authors show that it is easy to grossly overestimate the of cumulative measures over a person's entire career.

The study analyzed 762 scientists from three disciplines: physics, biology, and mathematics. By applying future impact models to these careers, a number of subtle, but critical, flaws in current models were identified. Specifically, the 'h-index' contains false autocorrelation, resulting in a significant overestimation of "predictive power". Moreover, the "predictive power" of these models vary greatly with the career age of scientists, producing least accurate estimates for already risk-burdened early career researchers.

Aalto researchers concluded that care must be taken to select the correct measures and methods to evaluate scientific candidates in the future. Increased attention should be paid to the potential shortfalls of quantitative methods when applied to the decision-making process.

The results were published in the Scientific Reports, a research publication from the publishers of Nature.

Explore further: Researchers use science to predict success

More information: Penner, O. et al. On the Predictability of Future Impact in Science, Scientific Reports, 2013. e-print; arXiv:1306.0114

add to favorites email to friend print save as pdf

Related Stories

Researchers use science to predict success

Oct 07, 2013

We all want to know the secret to success and physicists are no different. Like the rest of the academic community, physicists rely on various quantitative factors to determine whether a researcher will enjoy ...

Novel methods to support decision making processes

Jan 24, 2013

Mathematical analysis based on numbers and numerical estimates is widely used in decision making everywhere from public administration and business to environmental conservation. Methods for decision analysis evaluate different ...

New IT tool predicts book sales prior to publication

Jun 25, 2013

In Granada, researchers have developed a new IT system that predicts the volume of sales a book will have if it is finally published. This information will be of great value to publishers, who could use it to decide on the ...

New formula predicts if scientists will be stars

Sep 12, 2012

A medical school committee is weighing whether to hire a promising young neuroscientist. Will she have a brilliant future as a researcher, publish in top journals and nab abundant research funds?

Recommended for you

Scholar tracks the changing world of gay sexuality

Sep 19, 2014

With same-sex marriage now legalized in 19 states and laws making it impossible to ban homosexuals from serving in the military, gay, lesbian and bisexual people are now enjoying more freedoms and rights than ever before.

User comments : 0