Studies of scientific bias targeting the right problems, study finds

March 20, 2017
Credit: Petr Kratochvil/Public Domain

In all fields of science, small studies, early studies and highly cited studies consistently overestimate effect size, according to a study led by researchers at the Stanford University School of Medicine.

A scientist's early career status, isolation from other researchers and involvement in misconduct also appeared to be risk factors for unreliable results, the research team reported.

A paper describing the work will be published online March 20 in Proceedings of the National Academy of Sciences. The lead author is Stanford senior research scientist Daniele Fanelli, PhD, and the senior author is John Ioannidis MD, DSc, professor of medicine and of health research and policy.

Virtually all scientific work may be afflicted by some kind of , Ioannidis said. But how common different kinds of bias are, what factors cause bias, which kinds of bias are most common in different disciplines and how bias can be reduced are all questions being examined by researchers who study how science is done.

"I think that this is a mapping exercise," said Ioannidis. "It maps all the main biases that have been proposed across all 22 scientific disciplines. Now we have a map for each scientific discipline, which biases are important and which have a bigger impact, and therefore scientists can think about where do they want to go next with their field."

To show which sources of scientific bias were most common, the researchers reviewed more than 3,000 meta analyses that included nearly 50,000 individual research studies across 22 scientific fields.

"Our study tested with much greater accuracy than before several hypotheses about the prevalence and causes of bias," said Fanelli. "Our results send a reassuring message, but only in part."

Types of bias

The researchers examined seven hypothesized kinds of scientific bias:

  • Small-study effect: when studies with small sample sizes report large effect sizes.
  • Gray literature bias: the tendency of smaller or statistically insignificant effects to be reported in PhD theses, conference proceedings or personal communications rather than in peer-reviewed literature.
  • Early-extremes effect: when extreme or controversial findings are published early just because they are astonishing.
  • Decline effect: when reports of extreme effects are followed by subsequent reports of reduced effects.
  • Citation bias: the larger the effect size, the more likely the study will be cited.
  • United States-effect: when U.S. researchers overestimate effect sizes.
  • Industry bias: when industry sponsorship and affiliation affect the direction and size of reported effects.

The Stanford team also looked at factors that have been hypothesized to increase the risk of bias, such as size and types of collaborations, the gender of the researchers or pressure to publish.

By far, the greatest bias came from small studies, while other sources of bias had relatively small effects. The team also found that small and highly cited studies and those in peer-reviewed journals seemed more likely to overestimate effects; U.S. studies and early studies seemed to report more-extreme effects; early-career researchers and researchers working in small or long-distance collaborations were more likely to overestimate effect sizes; and, not surprisingly, researchers with a history of misconduct tended to overestimate effect sizes.

On the other hand, studies by highly cited authors who published frequently were not more affected by bias than average. Research by men was no more likely to show bias than that of women. And scientists in countries with very strong incentives to publish, such as the United States, didn't seem to have more bias than studies from countries where the pressure was less. These results confirm, with much greater accuracy, previous studies on retractions and corrections and studies using more indirect proxies of bias.

"A country that has incentives to publish more may also have other features that make its science better," Ioannidis said.

Each kind of bias may result from a variety of mechanisms. For example, in small studies, it's easier to get "statistical significance" if the effect size is large. Since studies with statistically significant results are more likely to be published, it follows that small studies with large effects are also more likely to be published. Even independent of statistical significance, larger effects are more likely to be published than smaller effects.

Ioannidis said that in the data they examined, the influence of different kinds of bias changed over time and seemed to depend on the individual scientist. "We show that some of the patterns and risk factors seem to be getting worse in intensity over time," he said. "This is particularly driven by the social sciences, so if you broke scientific fields into big bins of biology, medicine, physical sciences and social sciences, it seems that the social sciences are seeing the more prominent worsening of these biases over time."

One of the most unexpected findings of the study, Fanelli said, was that the kinds and amounts of bias were very irregularly distributed across the literature. "Although bias may be worryingly high in specific research areas, it is nonexistent in many others," he said. "So bias does not undermine the scientific enterprise as a whole."

Another finding of the study is that the relative magnitude of biases closely reflects the level of attention that they receive in the literature. That is, the kinds of biases researchers are most concerned about are in fact the ones they should be concerned by. "Our understanding of bias is improving and our priorities are set on the right targets," said Fanelli.

But that's no cause for complacency, he said. "We perhaps understand bias better, but we are far from having rid science of it. Indeed, our results suggest that the challenge might be greater than many think because interventions might need to be tailored to the needs and problems of individual disciplines of fields. One-size-fits all solutions are unlikely to work."

Solutions and interventions

Ioannidis likewise cautioned that the data are purely observational, not experimental, and the question of how to reduce bias is far from clear. For example, he said, just because small studies tend to give exaggerated results doesn't mean we should stop doing them. "One might say immediately, well, we need to do large studies," he said. "That would be an intervention. But you can't necessarily translate an association directly into an effective intervention."

"I think that one can take each one of these biases and say, 'Well, let's try to reduce it or eliminate it,'" he added. "Some of them are easier to reduce or eliminate, but then we have to see what that does to the wider scientific literature."

That said, Ioannidis said he believes that many fields would benefit from having both larger studies and the involvement of numerous authors who are close enough to monitor one another's work. "But that's something that is an extrapolation," he said.

Making science better is quite possible, he said. "Physics, for example, at some point decided that they've had enough of these tiny studies done by small teams, and they went for a multi-team collaborative model," he said. "That changed the entire paradigm of how they do research. It probably largely removed the problem of small-study bias."

When physicists work together on huge projects, said Ioannidis, "you don't have the problem of these thousands and tens of thousands of physicists, each one of them running their tiny experiment, and then just waiting to collate the results after the fact and trying to make sense of them."

Since each field of science has a slightly different profile of biases, it makes sense, he said, for each one to choose the best ways to reduce the biases afflicting their field.

"This has to be a grass-roots movement," Ioannidis said. "It has to be something that scientists believe is good for their science to do. Top-down approaches, such as institutions and funding agencies trying to promote best practices, could also help, but it has to be an agreement, and an agreement among all these stakeholders. And obviously, scientists need to believe that this is something that will help the results and their to be more reliable."

Explore further: Industry funding biases drug trial studies in favor of sponsors' products

More information: Meta-assessment of bias in science, Proceedings of the National Academy of Sciences,
www.pnas.org/cgi/doi/10.1073/pnas.1618569114

Related Stories

Rating systems may discriminate against Uber drivers

December 16, 2016

Many new "sharing economy" companies, like Uber and Airbnb, use consumer-sourced ratings to evaluate their workers – but these systems can be fraught with difficulties, including bias based on race or gender.

Bias pervades the scientific reporting of animal studies

July 17, 2013

A new study published in the open access journal PLOS Biology suggests that the scientific literature could be compromised by substantial bias in the reporting of animal studies, and may be giving a misleading picture of ...

How do men and women respond to gender bias in STEM?

January 8, 2015

Research has revealed that gender biases limit the opportunities for women within science, technology, engineering, and mathematics (STEM) disciplines. But just how prevalent are these biases and how are they perceived differently ...

Recommended for you

UN says world population will reach 9.8 billion in 2050

June 22, 2017

India's population is expected to surpass China's in about seven years and Nigeria is projected to overtake the United States and become the third most populous country in the world shortly before 2050, a U.N. report said ...

Authenticity key to landing a new job

June 22, 2017

At job interviews, relax and be yourself - if you're good, being yourself may be the best way to secure a job offer, according to a new study involving UCL researchers.

17 comments

Adjust slider to filter visible comments by rank

Display comments: newest first

ab3a
1 / 5 (3) Mar 20, 2017
They left out another bias, and it is significant:

Government funding bias. This is a variant of industry bias. However just as industry bias can tend to yield results that the industry would like to see, Government will yield results that governments would like to see. In other words, an excuse to legislate and regulate more.
Whydening Gyre
5 / 5 (2) Mar 20, 2017
Pretty simple resolution - just dial it back by a half to a third...
Da Schneib
4.2 / 5 (5) Mar 20, 2017
I have a better idea.

Congress should give money without any restrictions on it for scientific research to reputable researchers, and Congress is not competent to judge their reputability so they should rely on other scientists to judge it.

Simple as that.

Gee, don't we already do that? You know, the National Science Foundation and stuff?

Just askin'.
Da Schneib
4.4 / 5 (7) Mar 20, 2017
@ab3a, you are engaging in Motivated Reasoning, a well-known psychological reasoning flaw.

You do the same with your climate denial. It's pretty obvious you're not familiar with modern psychology and are engaging in classical reasoning flaws. I suggest you study the literature and find out why you are wrong.
Captain Stumpy
4 / 5 (4) Mar 21, 2017
Government will yield results that governments would like to see. In other words, an excuse to legislate and regulate more
@ab3a
your conspiracist ideation is showing...
http://journals.p....0075637

so... quick question: what if everyone is seeing the same thing regardless of gov't affiliation, nation, culture, beliefs, or location?
One would think that, due to the inability of cultures to agree with each other, if they're all coming up with the exact same conclusions then there is something known as a scientific fact...

they're validating the studies that came before them

so you're choosing to ignore the entire planet and the science results from everywhere because ...????

https://phys.org/...ies.html
ab3a
1 / 5 (2) Mar 21, 2017
You see nothing wrong with government sponsored research but are clearly suspicious of industry research. Therefore I must be wrong with my "motivated reasoning"?

They are both large organizations, albeit with different motivations. Why is one to be suspected, but the other is not?
antialias_physorg
5 / 5 (5) Mar 21, 2017
Industry sponsored research deals in outcomes. Industry does not want a study that doesn't lead to something they can't use. If they can't make a product out of it they will at the very least want to cite the data in some advertising blurb.

Government doesn't push for outcomes. Government grants are perfectly fine if a study yields no significant results but is scientifically sound.
And make no mistake: such studies are worthwhile. E.g. knowing that drug X has no effect is as good as knowing that drug X has an effect - it prevents people getting duped into buying drug X.
Captain Stumpy
3.7 / 5 (3) Mar 21, 2017
You see nothing wrong with government sponsored research but are clearly suspicious of industry research. Therefore I must be wrong with my "motivated reasoning"?
@ab3a
1- that's called transference. try re-reading what i wrote because i never mentioned me being suspicious of anything

2- see AA_P above

3- you've typically been against anything that is specific to AGW, hence my argument above

that means, by definition, that you've made a choice to ignore the scientific results from not only every other gov't on the planet, but also every civilian non-gov't sponsored research on the planet as well

so, you've chosen to consider all the research on a specific topic as part of a conspiracy simply because you don't want to accept the research

that is called conspiracist ideation and is no different than the argument xtians use that their beliefs be accepted as equal to science
Da Schneib
5 / 5 (2) Mar 21, 2017
You see nothing wrong with government sponsored research but are clearly suspicious of industry research.
I am suspicious because all the government sponsored research, by governments that are at odds, is the same, and different from all the industry research, by companies that are in agreement.

Your ability to detect unwitting conspiracies is in question, and appears to be denigrated.

Conspiracy theories are the last resort of the hard of thinking.
ab3a
1 / 5 (3) Mar 21, 2017
@Stumpy, my opinions on global warming are not relevant here. In any case, if you read what I have written, I do not dispute that the climate is warming. I dispute the notion that we should pay for carbon dioxide indulgences to keep global warming under control. That's not the same thing.

In other words, you're misrepresenting my positions on orthogonal subjects and then you fail to answer my question. Why do you suspect industry but not government sponsored research? I'm not passing judgement here, I'm asking what I believe to be a very civil, decent question.

At the end of the day I suspect this comes down to political views. You would prefer to trust a government over a profit oriented enterprise. You can declare that and I can accept it even though I disagree with it.

Instead you squirm and dodge the question. I'm not impressed.
Da Schneib
5 / 5 (4) Mar 21, 2017
my opinions on global warming are not relevant here.
Sure they are.

Especially when you make the same logical fallacy over and over again.
Captain Stumpy
3 / 5 (2) Mar 22, 2017
@Stumpy, my opinions on global warming are not relevant here
@ab3a
they are when you continue to reiterate the same logic fails

it's representative of your bias etc - establishes a precedent
In any case, if you read what I have written
you mean like you do?

see point 1 in last post - i don't suspect anything, yet you continue to make that a point of argument
you're misrepresenting my positions on orthogonal subjects and then you fail to answer my question
LOL
again, see point 1 above

also - i've not misrepresented anything and i did answer your post, you just don't like what i said

and might i point out that you also didn't answer my first question...
You would prefer to trust a government over a profit oriented enterprise
wrong
i trust neither

i trust only what can be validated
period
(you know...like the scientific method)

to share your own words back:
... you squirm and dodge the question. I'm not impressed.
antialias_physorg
5 / 5 (5) Mar 22, 2017
At the end of the day I suspect this comes down to political views. You would prefer to trust a government over a profit oriented enterprise.

Just ask yourself: how often has self-commitment by the industry to increase quality, lower waste, etc. actually worked? Hint: the answer starts with 'n' and ends with 'ever'.
And these are the people you trust? That seems somewhat crazy.

Changes have only ever come about after being mandated by law.
And no: government isn't perfect or a perfect solution to everything. But profit oriented businesses are only good at being profit oriented - and nothing else. Wherever the issue is other than 'immediate profit' businesses aren't the ones you should go to. And this most definitely includes science.
ab3a
not rated yet Mar 22, 2017
Changes have only ever come about after being mandated by law.


If you say that, then you must believe that most of the technological progress humanity has made came at the behest of government. I disagree with that.

In fact, I don't trust any large organization, not industry, and not government. So I don't understand why you three give government funding a free pass for research bias concerns whereas you don't give that same free pass to industry.

For me, and you're welcome to disagree, I worry that, like industry, government also corrupts and biases research. Clearly the motives and means are different. But to suggest that this isn't much of a problem and shouldn't be watched, is foolish.

As further proof that such thing do happen, see some of the research in to genetics and its application to eugenics by governments such as Soviet Russia. Think that something similar can't happen here? I have a bridge to Brooklyn for sale...
antialias_physorg
5 / 5 (4) Mar 22, 2017
If you say that, then you must believe that most of the technological progress humanity has made came at the behest of government.

No. But technological progress - especially today where things have gotten very subtle and complicated - comes on the back of solid science. The days of cobbling together something in your garage that changes the world are over. Even for those inventors they are reliant on computers and hi tech which again was developed only after someone did the hard science on this.

So I don't understand why you three give government funding a free pass for research bias concerns whereas you don't give that same free pass to industry.

Easy.
Industry decides what to fund based on profitability concerns. This introduces bias.
Government grants are NOT decided by someone with an agenda what to fund. The funding is decided by panels of scientists. Since scientists' interests is science there is no (or at the very least much less) bias
antialias_physorg
5 / 5 (4) Mar 22, 2017
As further proof that such thing do happen, see some of the research in to genetics and its application to eugenics by governments such as Soviet Russia.

Good point. That is when individuals who aren't scientists (e.g. politicians) push an agenda.
But that is not how science and grants is set up in most of the world.

Yes, government could start to take over (of course the result is bad science. Because any time you go into research with preconceived notions of what the result should be you get biased science). It's up to the voters to vote for people who don't instrumentalize science. They didn't this time in the US and we'll see how much damage will be done.
Captain Stumpy
3 / 5 (2) Mar 22, 2017
Changes have only ever come about after being mandated by law
If you say that, then you must believe that most of the technological progress humanity has made came at the behest of government
ab3a
uhm... that's nonsensical IMHO - and that isn't what was said or even inferred, from what i can tell
i don't think anyone agrees with that
For me, and you're welcome to disagree, I worry that, like industry, government also corrupts and biases research. Clearly the motives and means are different. But to suggest that this isn't much of a problem and shouldn't be watched, is foolish
no one is saying it shouldn't be watched, so that is also nonsensical to me

it boils down to validation though, WRT gov't funded or industrial research - which is how science works

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.