Criteria for funding and promotion lead to bad science

November 10, 2016, Public Library of Science
Wanderer, Caspar David Friedrich. Credit: Photographic reproduction by Cybershot800i. (Diff), Wikimedia Commons

Scientists are trained to carefully assess theories by designing good experiments and building on existing knowledge. But there is growing concern that too many research findings may in fact be false. New research publishing 10 November in open-access journal PLOS Biology by psychologists at the universities of Bristol and Exeter suggests that this may happen because of the criteria used in funding science and promoting scientists which, they say, place too much weight on novel, eye-catching findings.

Some scientists are becoming concerned that published results are inaccurate—a recent attempt by 270 scientists to reproduce the findings reported in 100 psychology studies the Reproducibility Project: Psychology found that only about 40 per cent could be reproduced.

This latest study shows that we shouldn't be surprised by this, because researchers are incentivised to work in a certain way if they want to further their careers, such as running a large number of small studies, rather than a smaller number of larger, more definitive ones. But while this might be good for their careers, it won't necessarily be good for .

Professor Marcus Munafò and Dr Andrew Higginson, researchers in psychology at the universities of Bristol and Exeter, concluded that scientists aiming to progress should carry out lots of small, exploratory studies because this is more likely to lead to surprising results. The most prestigious journals publish only highly novel findings, and scientists often win grants and get promotions if they manage to publish just one paper in these journals, which means that these small (but unreliable) studies may be disproportionately rewarded in the current system.

The authors used a mathematical model to predict how an optimal researcher who is trying to maximise the impact of their publications should spend their research time and effort. Scientific researchers have to decide what proportion of time to invest in looking for exciting new results rather than confirming previous findings. They also must decide how much resource to invest in each experiment.

The model shows that the best thing for career progression is carry out lots of small exploratory studies and no confirmatory ones. Even though each experiment is less likely to identify a real effect if it's there, they are likely to get some false positives, which unfortunately are often published too.

Dr Higginson said: "This is an important issue because so much money is wasted doing research from which the results can't be trusted; a significant finding might be just as likely to be a false positive as actually be measuring a real phenomenon."

This wouldn't happen if more publications, rather than one or two high profile ones, mattered to scientists' careers, nor if novel findings weren't prized so much more than confirmatory work that confirms previous findings, say the researchers.

So is there any way to overcome this problem of bad scientific practice? There could be immediate solutions, as Professor Munafò explained: "Journal editors and reviewers could be much stricter about good statistical procedures, such as insisting on large sample sizes and tougher statistical criteria for deciding whether an effect has been found."

There are already some encouraging signs - for example, a number of journals are introducing reporting checklists which require authors to state, among other things, how they decided on the sample size they used. Funders are also making similar changes to grant application procedures.

"The best thing for scientific progress would be a mixture of medium-sized exploratory studies with large confirmatory studies," said Dr Higginson. "Our work suggests that researchers would be more likely to do this if agencies and promotion committees rewarded asking important questions and good methodology, rather than surprising findings and exciting interpretations."

Explore further: The irreproducibility crisis – an opportunity to make science better

More information: Higginson AD, Munafò MR (2016) Current Incentives for Scientists Lead to Underpowered Studies with Erroneous Conclusions. PLoS Biol 14(11): e2000995. DOI: 10.1371/journal.pbio.2000995

Related Stories

Opinion: Why isn't science better? Look at career incentives

September 21, 2016

There are often substantial gaps between the idealized and actual versions of those people whose work involves providing a social good. Government officials are supposed to work for their constituents. Journalists are supposed ...

Recommended for you

17 comments

Adjust slider to filter visible comments by rank

Display comments: newest first

freeiam
2.8 / 5 (13) Nov 10, 2016
Next thing they find is that science is determined by culture and not necessarily by fact.
Take for example 'climate science'.
Chris_Reeve
Nov 10, 2016
This comment has been removed by a moderator.
Captain Stumpy
3 / 5 (8) Nov 10, 2016
for astrophysics and cosmology are operating at the very edge of what we consider "laboratory science"
@chris/alfven
1- repeating a lie doesn't make it more true - http://www.pppl.gov/

need i post links to other labs proving you wrong yet again?

2- theoretical anything always operates at the edge of what we know, this is where we get hypothesis from, which leads to testable experimentation, and therein lies the lab link used above

3- just because you don't understand the physics or why it debunks your religious fanatical acceptance of electric universe cult propaganda doesn't mean you're right about astrophysics/cosmology in general
with time and effort, you , too, can learn about reality: https://ocw.mit.edu/index.htm

4- honesty means accepting evidence based changes - that is the basis of the scientific method

you, however, don't accept evidence - why?
http://journals.p....0075637

arcmetal
3 / 5 (2) Nov 11, 2016
At some point in the future a similar article will be written whereas instead of "social science", the topic will be "big bang theory".
antialias_physorg
3.7 / 5 (3) Nov 11, 2016
In psychology I'm not really suprised about the results. Humans are complex, and studies often have very low statistical power.
"This is an important issue because so much money is wasted doing research from which the results can't be trusted;

The alternative is, of course, to only make large, well funded studies. Yeah...like that's going to happen (especially the funding part).
Funding agencies don't like to take risks. And the risk with any study (even a large one) is: no results. That's just intrinsic to science - you can never guarantee results because you're working on the unknown.

But in the case of 'no results' in that case heads will roll. Better (from a political POV) to fund many small studies, and get some results.

Money-politics. Gotta love it :-/
hawkingsbrother
Nov 11, 2016
This comment has been removed by a moderator.
hawkingsbrother
Nov 11, 2016
This comment has been removed by a moderator.
ab3a
5 / 5 (1) Nov 11, 2016
To be a science, there has to be a theory, an hypothesis, and a controlled experiment that yields consistent and reproducible results. This is why reproducibility is such a big deal.

This is why Richard Feynman argued that the social sciences are not sciences. They are studies, but they are not sciences. This is why he also argued that studying weather is not a science. Even studies such as cosmology and astronomy, while they are acts of discovery, are not science. Controlled experiments are not possible in these studies.

On the other hand, chemistry, physics, and most biological experiments are entirely reproducible. Feynman deemed these to be sciences. So if funding such experiments in one manner or another affects the outcome, then it isn't science. It is deception.

antialias_physorg
2.3 / 5 (3) Nov 11, 2016
Controlled experiments are not possible in these studies.

In all these you can make predictions of - as yet - unobserved phenomena. That's pretty scientific in my book.
Unscientific would be something like history where you can make no predictions but simply fit theories to observables (as there are no unobservables. What is known of history is known and new discoveries of historic events are exceedingly rare...and then they are never predicted by any historic theory)...When applying history to political sciences then the picture changes somewhat -as here you can make future predictions.

So if funding such experiments in one manner or another affects the outcome

Better funding - more solid data base (and hence better scientific value).
xponen
5 / 5 (1) Nov 11, 2016
@ab3a
Biological experiments is not immune to irreproducibility. There's a scandal where cell lines sold by a company don't actually react consistently to chemicals, making tons of experiment irreproducible.

Physic experiment is also not immune. A notable example is the Cold fusion experiment.

I think mainstream science is mostly not effected by this issue because bad experiments usually don't have follow up. Even Einstein himself is remarked by his peer as "..publishing paper that falsify his previous paper", meaning; a follow up and a maturation is a sign of legitimacy, which don't exist for bad papers.
ab3a
5 / 5 (1) Nov 11, 2016
@antialias_physorg: I can make a prediction of who is going to win the next ten football games, and I may even be correct. However that does not make it a science. The aspect that defines science is the ability to consistently test a theory through controlled experiments.

@xponen: Irreproducibility is the result of deception. In a surprising number of cases it is self deception. There may not even be malice involved. Read the history of the effort to determine the charge of an electron with the Millikan oil drop experiment. However, the fact that an experiment is not well controlled does not invalidate the label of science through the scientific process.
mtnphot
not rated yet Nov 11, 2016
The scientific method calls for an hypothesis and testing. However there are aspects of science that are more descriptive. For example the original discovery of the red spot of Jupiter. No one set up a hypothesis that there would be a red spot there and then went to look for it. It was discoverd by pure obsrvation. However the observation was confirmed by others observing it as well.
Some of the most important discoveries have been made by pure observation of unexpected results; however once the discovery was originally made, the scientific method comes into play to confirm and explain the original observation.
antigoracle
5 / 5 (2) Nov 11, 2016
If Climate "Science" was science, then we need look no further for conclusive proof of this research.
antialias_physorg
not rated yet Nov 14, 2016
I can make a prediction of who is going to win the next ten football games, and I may even be correct. However that does not make it a science.

Of course the predictions have to match the (then) unobserved phenomena when they are eventually observed. I took that as a given. If just the prediction mattered then astrology would be a science.

Note that there are such works out there. Example which uses a neural network to predict NFL and NCAA outcomes:
http://personal.d...11/4.pdf
ab3a
5 / 5 (1) Nov 14, 2016
@antialias_physorg Whether a study is a science is not is not predicated on whether you can predict what something will do, but that the something can be controlled and tested independently. We can not control and test each player to know the outcome of a football game. Even if you could arrive at a neural net model that inputs parameters from coaching staff, player performance, which stadium, the weather, and so on, this is still not a science.

In other words, predictability via models does not imply that it is science. The difference between a science and a study is the controlled experiment. If you have one is it is a science. If you do not, it is not.
antialias_physorg
5 / 5 (1) Nov 14, 2016
We can not control and test each player to know the outcome of a football game.

Read the linked paper.


In other words, predictability via models does not imply that it is science.

If a theory perdicts e.g. the existence of magnetars (while being consistent with all observations to date) and a magnetar is found then - even though that wasn't a controlled experiment - is scientific.
ab3a
not rated yet Nov 15, 2016
If a theory perdicts e.g. the existence of magnetars (while being consistent with all observations to date) and a magnetar is found then - even though that wasn't a controlled experiment - is scientific.


Scientific? Maybe, but it is still not a science. Astronomy and Cosmology are not sciences. They are studies that use scientific theories derived from controlled experiments, and theories derived from studies and observations. Discovery is not a controlled experiment.

Many wish to cloak themselves in the certainty of science. So they claim the title of a science even though there is very little about the study that is actually scientific. For example, Meteorology is not a science. The study of human population changes and geography is not a science ("social science"). Cosmology is not a science. And some even argue that string theory, if it can not be tested, is not a science either. These are important studies, but they're not science.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.