We need to talk about the bad science being funded

We need to talk about the bad science being funded
Good science loses out when bad science gets the funding. Credit: Shutterstock/Looker Studio

Spectacular failures to replicate key scientific findings have been documented of late, particularly in biology, psychology and medicine.

A on the issue, published in Nature this May, found that about 90% of some 1,576 researchers surveyed now believe there is a reproducibility crisis in science.

While this rightly tarnishes the public belief in science, it also has serious consequences for governments and philanthropic agencies that fund research, as well as the pharmaceutical and biotechnology sectors. It means they could be wasting billions of dollars on research each year.

One contributing factor is easily identified. It is the high rate of so-called false discoveries in the literature. They are false-positive findings and lead to the erroneous perception that a definitive scientific discovery has been made.

This high rate occurs because the studies that are published often have low statistical power to identify a genuine discovery when it is there, and the effects being sought are often small.

Further, dubious scientific practices boost the chance of finding a statistically significant result, usually at a probability of less than one in 20. In fact, our probability threshold for acceptance of a discovery should be more stringent, just as it is for discoveries of new particles in physics.

The English mathematician and the father of computing Charles Babbage noted the problem in his 1830 book Reflections on the Decline of Science in England, and on Some of Its Causes. He formally split these practices into "hoaxing, forging, trimming and cooking".

'Trimming and cooking' the data today

In the current jargon, trimming and cooking include failing to report all the data, all the experimental conditions, all the statistics and reworking the probabilities until they appear significant.

The frequency of many of these indefensible practices is above 50%, as reported by scientists themselves when they are given some incentive for telling the truth.

The English philosopher Francis Bacon wrote almost 400 years ago that we are influenced more by affirmation than negatives and added:

Man prefers to believe what he prefers to be true.

Deep-seated cognitive biases, consciously and unconsciously, drive scientific corner-cutting in the name of discovery.

This includes fiddling the primary hypothesis being tested after knowing the actual results or fiddling the statistical tests, the data or both until a statistically significant result is found. Such practices are common.

Even large randomised controlled clinical trials published in the leading medical journals are affected (see compare-trials.org) – despite research plans being specified and registered before the trial starts.

Researchers rarely stick exactly to the plans (about 15% do). Instead, they commonly remove registered planned outcomes (which are presumably negative) and add unregistered ones (which are presumably positive).

Publish or perish

We do not need to look far to expose the fundamental cause for the problematic practices pervading many of the sciences. The "publish or perish" mantra says it all.

Academic progression is hindered by failure to publish in the journals controlled by peers, while it is enhanced by frequent publication of, nearly always positive, research findings. Does this sort of competitive selection sound familiar?

It is a form of cultural selection – natural, in that it is embedded in the modern culture of science, and selective in that only survivors progress. The parallels between biological and selection related to culture have long been accepted. Charles Darwin even described its role in development of language in his The Descent of Man (1871).

Starkly put, the rate of publication varies between scientists. Scientists who publish at a higher rate are preferentially selected for positions and promotions. Such scientists have "children" who establish new laboratories and continue the publication practices of the parent.

Good science suffers

In another study published in May, researchers modelled the intuitive but complex interactions between the pressure and effort to publish new findings and the need to replicate them to nail down true discoveries. It is a well-argued simulation of the operation and culture of modern science.

They also conclude that there is natural selection for bad scientific practice because of incentives that simply reward "publication quantity":

Scrupulous research on difficult problems may require years of intense work before yielding coherent, publishable results. If shallower work generating more publications is favored, then researchers interested in pursuing complex questions may find themselves without jobs, perhaps to the detriment of the scientific community more broadly.

The authors also reiterate the low power of many studies to find a phenomenon if it was truly there. Despite entreaties to increase statistical power, for example by collection of more observations, it has remained consistently low for the last 50 years.

In some fields, it averages only 20% to 30%. Natural academic selection has favoured publication of a result, rather than generation of new knowledge.

The impact of Darwinian selection among scientists is amplified when government support for science is low, growth in the scientific literature continues unabated, and universities produce an increasing number of PhD graduates in science.

We hold an idealised view that science is rarely fallible, particularly and . Yet many fields are filled with publications of low-powered studies with perhaps the majority being wrong.

This problem requires action from scientists, their teachers, their institutions and governments. We will not turn natural selection around but we need to put in place selection pressures for getting the right answer rather than simply published.


Explore further

Publish or perish culture encourages scientists to cut corners

Journal information: Nature

This article was originally published on The Conversation. Read the original article.
The Conversation

Citation: We need to talk about the bad science being funded (2016, July 19) retrieved 24 August 2019 from https://phys.org/news/2016-07-bad-science-funded.html
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.
26 shares

Feedback to editors

User comments

Jul 20, 2016
The problem is null hypothesis significance testing, which does not tell you want to know even if used properly. Notice (go figure!) that the author gets it wrong. One cannot reject the null and claim that there is less than a p of 0.05 that the null has been incorrectly rejected. The quantitative meaning of a p-value is predicated on the truth of the null. Guess what? It makes no sense to say, if the null is true, there is a 95% chance that it is false! Anyway, a solution is to abandon group designs altogether. So-called single-subject research is far superior, and generally includes both within- and between-subject replication within the research to be published. That is, the findings have already been replicated before the paper is submitted. In any event, NHST should be banned - not "policed" - banned!

Jul 20, 2016
Since the threshold is at p=0.05 it is only to be expected that up to 1 in 20 studies report something which isn't replicable. That's just the tradeoff you have in many fields where the amount of data is, by nature of the field, limited (psychology, pharmacology, basically anything to do with humans where univariate studies are next to impossible).
What the author wants here is to only fund research that reports true positives - and that#s just not how research works (or even can work).

Single subject design is superior but not practical in many gfields.
Extreme example: You can't do an A-B-A test when you're in a phase II cancer drug study and the drug shows efficacy. But even in other studies: humans change during a trial. You never have a clean test subject where *only* the tested variable changes.

https://en.wikipe...t_design

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more