Models to predict scientists' future impact often fail

Oct 30, 2013
Researchers at work.

Models universities partially use to forecast scientists' future contributions are not as reliable as previously thought. In a recent study, Aalto University and IMT Institute for Advanced Studies Lucca researchers demonstrate mathematical analysis of past performance cannot reliably determine the future performance. This means that current models dangerously overestimate the predictability and should not be used for career advancement decision process.

"Based on our results, the predictability of current models for real application in recruitment decisions is questionable. Efforts to model future impact need to be aimed more directly at applications in the career advancement decision process," says Professor Santo Fortunato from Aalto University in Finland.

In recent years it has become more common for universities to use quantitative measures for yard-sticking the productivity and impact of individual researchers to help reduce hiring risks. Models thought to be capable of foreseeing a scientist's future impact by way of his or her future 'h-index' have become a common tool in recruitment decisions as well as other scientific evaluation, advancement, and reward processes.

Scientists concluded that great caution should be taken when attempting to forecast an individual's future based on their 'h-index'. Specifically the authors show that it is easy to grossly overestimate the of cumulative measures over a person's entire career.

The study analyzed 762 scientists from three disciplines: physics, biology, and mathematics. By applying future impact models to these careers, a number of subtle, but critical, flaws in current models were identified. Specifically, the 'h-index' contains false autocorrelation, resulting in a significant overestimation of "predictive power". Moreover, the "predictive power" of these models vary greatly with the career age of scientists, producing least accurate estimates for already risk-burdened early career researchers.

Aalto researchers concluded that care must be taken to select the correct measures and methods to evaluate scientific candidates in the future. Increased attention should be paid to the potential shortfalls of quantitative methods when applied to the decision-making process.

The results were published in the Scientific Reports, a research publication from the publishers of Nature.

Explore further: Researchers use science to predict success

More information: Penner, O. et al. On the Predictability of Future Impact in Science, Scientific Reports, 2013. e-print; arXiv:1306.0114

add to favorites email to friend print save as pdf

Related Stories

Researchers use science to predict success

Oct 07, 2013

We all want to know the secret to success and physicists are no different. Like the rest of the academic community, physicists rely on various quantitative factors to determine whether a researcher will enjoy ...

Novel methods to support decision making processes

Jan 24, 2013

Mathematical analysis based on numbers and numerical estimates is widely used in decision making everywhere from public administration and business to environmental conservation. Methods for decision analysis evaluate different ...

New IT tool predicts book sales prior to publication

Jun 25, 2013

In Granada, researchers have developed a new IT system that predicts the volume of sales a book will have if it is finally published. This information will be of great value to publishers, who could use it to decide on the ...

New formula predicts if scientists will be stars

Sep 12, 2012

A medical school committee is weighing whether to hire a promising young neuroscientist. Will she have a brilliant future as a researcher, publish in top journals and nab abundant research funds?

Recommended for you

Precarious work schedules common among younger workers

13 hours ago

One wish many workers may have this Labor Day is for more control and predictability of their work schedules. A new report finds that unpredictability is widespread in many workers' schedules—one reason ...

Girls got game

14 hours ago

Debi Taylor has worked in everything from construction development to IT, and is well and truly socialised into male-dominated workplaces. So when she found herself the only female in her game development ...

Computer games give a boost to English

Aug 28, 2014

If you want to make a mark in the world of computer games you had better have a good English vocabulary. It has now also been scientifically proven that someone who is good at computer games has a larger ...

Saddam Hussein—a sincere dictator?

Aug 28, 2014

Are political speeches manipulative and strategic? They could be – when politicians say one thing in public, and privately believe something else, political scientists say. Saddam Hussein's legacy of recording private discussions ...

User comments : 0