Does 'publication bias' affect the 'canonization' of facts in science?

December 20, 2016, University of Washington
The likelihood of a hypothesis being true or false moves up and down a 'ladder' as more and more experiments on the hypothesis are published. The probability that the scientific community considers the hypothesis to be true can become so high that researchers do not study the hypothesis any further - it will be taken for granted and perceived as a fact. The model shows that you have to publish a certain percentage of negative results (often 20-30 %) in order to ensure that hypotheses that are false do not end up being regarded as facts. Credit: Nissen, NBI and Bergstrom, UW

Researchers model how 'publication bias' does—and doesn't—affect the 'canonization' of facts in science

Arguing in a Boston courtroom in 1770, John Adams famously pronounced, "Facts are stubborn things," which cannot be altered by "our wishes, our inclinations or the dictates of our passion."

But facts, however stubborn, must pass through the trials of human perception before being acknowledged—or "canonized"—as facts. Given this, some may be forgiven for looking at passionate debates over the color of a dress and wondering if facts are up to the challenge.

Carl Bergstrom believes facts stand a fighting chance, especially if science has their back. A professor of biology at the University of Washington, he has used mathematical modeling to investigate the practice of science, and how science could be shaped by the biases and incentives inherent to human institutions.

"Science is a process of revealing facts through experimentation," said Bergstrom. "But science is also a human endeavor, built on human institutions. Scientists seek status and respond to incentives just like anyone else does. So it is worth asking—with precise, answerable questions—if, when and how these incentives affect the practice of science."

In an article published Dec. 20 in the journal eLife, Bergstrom and co-authors present a mathematical model that explores whether "publication bias"—the tendency of journals to publish mostly positive experimental results—influences how scientists canonize facts. Their results offer a warning that sharing positive results comes with the risk that a false claim could be canonized as fact. But their findings also offer hope by suggesting that simple changes to publication practices can minimize the risk of false canonization.

These issues have become particularly relevant over the past decade, as prominent articles have questioned the reproducibility of scientific experiments—a hallmark of validity for discoveries made using the scientific method. But neither Bergstrom nor most of the scientists engaged in these debates are questioning the validity of heavily studied and thoroughly demonstrated scientific truths, such as evolution, anthropogenic climate change or the general safety of vaccination.

"We're modeling the chances of 'false canonization' of facts on lower levels of the scientific method," said Bergstrom. "Evolution happens, and explains the diversity of life. Climate change is real. But we wanted to model if publication bias increases the risk of false canonization at the lowest levels of fact acquisition."

Bergstrom cites a historical example of false canonization in science that lies close to our hearts—or specifically, below them. Biologists once postulated that bacteria caused stomach ulcers. But in the 1950s, gastroenterologist E.D. Palmer reported evidence that bacteria could not survive in the human gut.

"These findings, supported by the efficacy of antacids, supported the alternative 'chemical theory of ulcer development,' which was subsequently canonized," said Bergstrom. "The problem was that Palmer was using experimental protocols that would not have detected Helicobacter pylori, the bacteria that we know today causes ulcers. It took about a half century to correct this falsehood."

The figure shows a false negative rate of 0.2 (which is the standard that many fields of science use). The standard for the false positive rate is 0.05 (the white/yellow line), but there are indications that it is higher in practice. The figure shows that if you publish less than 20-30 percent negative results, there will be a high probability that a false hypothesis will be perceived as being a fact after repeated experiments and published articles. The scientific community only avoid this if they publish a greater proportion of negative results. Credit: Nissen, NBI and Bergstrom, UW

While the idea of false canonization itself may cause dyspepsia, Bergstrom and his team—lead author Silas Nissen of the Niels Bohr Institute in Denmark and co-authors Kevin Gross of North Carolina State University and UW undergraduate student Tali Magidson—set out to model the risks of false canonization given the fact that scientists have incentives to publish only their best, positive results. The so-called "negative results," which show no clear, definitive conclusions or simply do not affirm a hypothesis, are much less likely to be published in peer-reviewed journals.

"The net effect of publication bias is that negative results are less likely to be seen, read and processed by scientific peers," said Bergstrom. "Is this misleading the canonization process?"

For their model, Bergstrom's team incorporated variables such as the rates of error in experiments, how much evidence is needed to canonize a claim as fact and the frequency with which negative results are published. Their mathematical model showed that the lower the publication rate is for negative results, the higher the risk for false canonization. And according to their model, one possible solution—raising the bar for canonization—didn't help alleviate this risk.

"It turns out that requiring more evidence before canonizing a claim as fact did not help," said Bergstrom. "Instead, our model showed that you need to publish more negative results—at least more than we probably are now."

Since most negative results live out their obscurity in the pages of laboratory notebooks, it is difficult to quantify the ratio that are published. But clinical trials, which must be registered with the U.S. Food and Drug Administration before they begin, offer a window into how often negative results make it into the peer-reviewed literature. A 2008 analysis of 74 clinical trials for antidepressant drugs showed that scarcely more than 10 percent of negative results were published, compared to over 90 percent for positive results.

"Negative results are probably published at different rates in other fields of science," said Bergstrom. "And new options today, such as self-publishing papers online and the rise of journals that accept some negative results, may affect this. But in general, we need to share negative results more than we are doing today."

Their model also indicated that negative results had the biggest impact as a claim approached the point of canonization. That finding may offer scientists an easy way to prevent false canonization.

"By more closely scrutinizing claims as they achieve broader acceptance, we could identify false claims and keep them from being canonized," said Bergstrom.

To Bergstrom, the model raises valid questions about how scientists choose to publish and share their findings—both positive and negative. He hopes that their findings pave the way for more detailed exploration of bias in scientific institutions, including the effects of funding sources and the different effects of incentives on different fields of science. But he believes a cultural shift is needed to avoid the risks of publication bias.

"As a community, we tend to say, 'Damn it, this didn't work, and I'm not going to write it up,'" said Bergstrom. "But I'd like scientists to reconsider that tendency, because science is only efficient if we publish a reasonable fraction of our negative findings."

Explore further: One reason so many scientific studies may be wrong

More information: Silas Boye Nissen et al, Publication bias and the canonization of false facts, eLife (2016). DOI: 10.7554/eLife.21451

Related Stories

False-positive mammography results are common

January 12, 2016

(HealthDay)—For women undergoing digital mammography screening, false-positive results are common, especially among younger women and those with risk factors, according to research published online Jan. 12 in the Annals ...

When finding 'nothing' means something

September 25, 2014

Scientists usually communicate their latest findings by publishing results as scientific papers in journals that are almost always accessible online (albeit often at a price), ensuring fast sharing of latest knowledge.

Recommended for you

Archaeologists discover Incan tomb in Peru

February 16, 2019

Peruvian archaeologists discovered an Incan tomb in the north of the country where an elite member of the pre-Columbian empire was buried, one of the investigators announced Friday.

Where is the universe hiding its missing mass?

February 15, 2019

Astronomers have spent decades looking for something that sounds like it would be hard to miss: about a third of the "normal" matter in the Universe. New results from NASA's Chandra X-ray Observatory may have helped them ...

What rising seas mean for local economies

February 15, 2019

Impacts from climate change are not always easy to see. But for many local businesses in coastal communities across the United States, the evidence is right outside their doors—or in their parking lots.

The friendly extortioner takes it all

February 15, 2019

Cooperating with other people makes many things easier. However, competition is also a characteristic aspect of our society. In their struggle for contracts and positions, people have to be more successful than their competitors ...


Adjust slider to filter visible comments by rank

Display comments: newest first

2 / 5 (4) Dec 20, 2016
The less negative feedback, the greater likelihood of self-delusion. Study would seem to validate the theory that the scientific consensus must always be wrong.
3.2 / 5 (13) Dec 20, 2016
It is CONSENSUS science that 80-95% of the Universe is composed of missing Dark Matter, this despite zero EVIDENCE or OBSERVATION which even hints at it's existence. Yeah, weave a smoothly written story filled with unbelievable fantasy & that becomes the most efficient modus operandi for passing it off to the gullible, for example those who will soon be casting their 1 Star downvotes for this post.
Dec 20, 2016
This comment has been removed by a moderator.
2 / 5 (4) Dec 21, 2016
"But neither Bergstrom nor most of the scientists engaged in these debates are questioning the validity of heavily studied and thoroughly demonstrated scientific truths, such as evolution, anthropogenic climate change or the general safety of vaccination."

Feeling the need to make this kind of statement is indicative of peer pressure bias to start with.

2 / 5 (4) Dec 21, 2016
Tom Wigley, a scientist at the University Corporation for Atmospheric Research, authored a Climategate 1.0 email asserting that his fellow Climategate scientists "must get rid of" the editor for a peer-reviewed science journal because he published some papers contradicting assertions of a global warming crisis.

AGWism is a cult.
2 / 5 (5) Dec 26, 2016
Experimentation and the publication should be the norm, otherwise bunk theories like einstein's GR get cannonised
not rated yet Dec 30, 2016
Short answer is 'yes'.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.