Models to predict scientists' future impact often fail

Oct 30, 2013
Researchers at work.

Models universities partially use to forecast scientists' future contributions are not as reliable as previously thought. In a recent study, Aalto University and IMT Institute for Advanced Studies Lucca researchers demonstrate mathematical analysis of past performance cannot reliably determine the future performance. This means that current models dangerously overestimate the predictability and should not be used for career advancement decision process.

"Based on our results, the predictability of current models for real application in recruitment decisions is questionable. Efforts to model future impact need to be aimed more directly at applications in the career advancement decision process," says Professor Santo Fortunato from Aalto University in Finland.

In recent years it has become more common for universities to use quantitative measures for yard-sticking the productivity and impact of individual researchers to help reduce hiring risks. Models thought to be capable of foreseeing a scientist's future impact by way of his or her future 'h-index' have become a common tool in recruitment decisions as well as other scientific evaluation, advancement, and reward processes.

Scientists concluded that great caution should be taken when attempting to forecast an individual's future based on their 'h-index'. Specifically the authors show that it is easy to grossly overestimate the of cumulative measures over a person's entire career.

The study analyzed 762 scientists from three disciplines: physics, biology, and mathematics. By applying future impact models to these careers, a number of subtle, but critical, flaws in current models were identified. Specifically, the 'h-index' contains false autocorrelation, resulting in a significant overestimation of "predictive power". Moreover, the "predictive power" of these models vary greatly with the career age of scientists, producing least accurate estimates for already risk-burdened early career researchers.

Aalto researchers concluded that care must be taken to select the correct measures and methods to evaluate scientific candidates in the future. Increased attention should be paid to the potential shortfalls of quantitative methods when applied to the decision-making process.

The results were published in the Scientific Reports, a research publication from the publishers of Nature.

Explore further: Researchers use science to predict success

More information: Penner, O. et al. On the Predictability of Future Impact in Science, Scientific Reports, 2013. e-print; arXiv:1306.0114

add to favorites email to friend print save as pdf

Related Stories

Researchers use science to predict success

Oct 07, 2013

We all want to know the secret to success and physicists are no different. Like the rest of the academic community, physicists rely on various quantitative factors to determine whether a researcher will enjoy ...

Novel methods to support decision making processes

Jan 24, 2013

Mathematical analysis based on numbers and numerical estimates is widely used in decision making everywhere from public administration and business to environmental conservation. Methods for decision analysis evaluate different ...

New IT tool predicts book sales prior to publication

Jun 25, 2013

In Granada, researchers have developed a new IT system that predicts the volume of sales a book will have if it is finally published. This information will be of great value to publishers, who could use it to decide on the ...

New formula predicts if scientists will be stars

Sep 12, 2012

A medical school committee is weighing whether to hire a promising young neuroscientist. Will she have a brilliant future as a researcher, publish in top journals and nab abundant research funds?

Recommended for you

Study examines use of GIS in policing

13 minutes ago

Police agencies are using Geographic Information Systems (GIS) for mapping crime, identifying crime "hot spots," assigning officers, and profiling offenders, but little research has been done about the effectiveness of the ...

When rulers can't understand the ruled

17 hours ago

Johns Hopkins University political scientists wanted to know if America's unelected officials have enough in common with the people they govern to understand them.

When casualties increased, war coverage became more negative

21 hours ago

As the number of U.S. casualties rose in Afghanistan, reporters filed more stories about the conflict and those articles grew increasingly negative about both the war effort and the military, according to a Penn State researcher. ...

User comments : 0