Study finds statistical error in large numbers of neuroscience papers

Sep 13, 2011 by Bob Yirka report

(PhysOrg.com) -- Sander Nieuwenhuis and his associates from the Netherlands have done a study on one particular type of statistical error that apparently crops up in an inordinately large number of papers published in neuroscience journals. In their paper, published in Nature Neuroscience, they claim that up to half of all papers published in such journals contain the error.

The problem lies in the way findings are presented. If a group of researchers, for example, applies something (chemical, food, energy, etc.) that is to cause an effect on something else (, populations, inanimate objects, etc.) and finds the amount of change caused by the main thing that is being studied is “significant” but the change in the control group is not, they cannot then, reasonably compare the two results and come up with something that they consider significant unless the differences between the two are actually statistically significant (based on additional research).

Why such errors appear in so many research papers is open to debate. Whether it’s due to researchers wishing to overstate their findings, ignorance, or simple sloppiness, it’s clear that more scrutiny and peer review must be done by researchers before submitting their work. Of course, that’s only half the equation, why are journals who obviously take their reputations very seriously not properly vetting such papers before publishing them?

In their study, the group reviewed 513 papers published in five different highly regarded journals over a two year period. They found half of the papers (where such an error was possible) had the error in them. In addition they also found that when looking at 120 articles published on (with cell and molecular themes) that 25 had the error in them.

Clearly there is a serious problem here; this research project highlights a problem that is likely present in other areas of science as well; namely the inaccuracies present in science journals, mainstream science magazines, the media and perhaps even in classroom lectures. Failing to check for and fix simple statistical inaccuracies in papers presenting results obtained in research, calls into question their very integrity.

Hopefully research studies such as this one will cause alarm both in the research and publishing communities and bring about better controls on both.

Explore further: 'Moral victories' might spare you from losing again

More information: Erroneous analyses of interactions in neuroscience: a problem of significance, Nature Neuroscience 14, 1105–1107 (2011) doi:10.1038/nn.2886

Abstract
In theory, a comparison of two experimental effects requires a statistical test on their difference. In practice, this comparison is often based on an incorrect procedure involving two separate tests in which researchers conclude that effects differ when one effect is significant (P < 0.05) but the other is not (P > 0.05). We reviewed 513 behavioral, systems and cognitive neuroscience articles in five top-ranking journals (Science, Nature, Nature Neuroscience, Neuron and The Journal of Neuroscience) and found that 78 used the correct procedure and 79 used the incorrect procedure. An additional analysis suggests that incorrect analyses of interactions are even more common in cellular and molecular neuroscience. We discuss scenarios in which the erroneous procedure is particularly beguiling.

via Guardian

Related Stories

Report: U.S. R&D publications decline

Nov 27, 2006

A science editor says the U.S. share of scientific papers published worldwide in peer-reviewed science and engineering journals is declining.

Science retracts cloning articles

Jan 13, 2006

The journal Science has retracted two articles by discredited South Korean scientist who claimed production of a stem-cell line from a cloned human embryo.

Recommended for you

P90X? Why consumers choose high-effort products

4 hours ago

Stuck in traffic? On hold for what seems like an eternity? Consumers often face situations that undermine their feelings of control. According to a new study in the Journal of Consumer Research, when a person's sense of con ...

Overdoing it: Multiple perspectives confuse consumers

4 hours ago

Television commercials for luxury vehicles pack a lot in their 30-second running times: the camera offers quick shots of the soft leather upholstery, the shiny colors, the state-of-the-art entertainment system, ...

User comments : 16

Adjust slider to filter visible comments by rank

Display comments: newest first

Nanobanano
2 / 5 (4) Sep 13, 2011
Lol.

Got so bad they write peer reviewed papers on the fallacies and inaccuracies in other peer reviewed papers, has it?
Vendicar_Decarian
2.3 / 5 (9) Sep 13, 2011
Obviously this is proof that the brain isn't he site of human cognition and that it lies in the spirit, just as the Bible says.

The conspiracy among these corrupt Neuro-scientists is being exposed, and their reign of socialist terror is coming to an end.

thales
5 / 5 (1) Sep 13, 2011
Obviously this is proof that the brain isn't he site of human cognition and that it lies in the spirit, just as the Bible says.

The conspiracy among these corrupt Neuro-scientists is being exposed, and their reign of socialist terror is coming to an end.



Is this real? Or is this a Poe?
abgough
5 / 5 (1) Sep 13, 2011
Obviously this is proof that the brain isn't he site of human cognition and that it lies in the spirit, just as the Bible says.

The conspiracy among these corrupt Neuro-scientists is being exposed, and their reign of socialist terror is coming to an end.



Is this real? Or is this a Poe?


Ha ha I thought the exact same thing. I hope this person is kidding. I believe there is a chance that human cognition may lie outside (or separately) from the brain itself, but that "theory" has nothing to do with the bible or a silly god.
Vendicar_Decarian
3 / 5 (6) Sep 13, 2011
It's all a scam by wine and cheese eating scientists who are out to steal my money.

These cushy scientists jobs and their porshes and disgusting lifestyles with interns and college children must be stopped.

Defund science now. All of it. It's the Devil's work.
Temple
5 / 5 (3) Sep 13, 2011
Science is self-correcting.

It's not error-proof. It's self-correcting. Here is but the latest in a long, *long* example of such cases where what was previously thought correct was rewritten or refined.

What many want to take from that is that we cannot trust anything Science tells us. That's unfortunate, as a great deal of 'mature' science has withstood uncountable attempts at proving it wrong or otherwise incomplete. Those cases may indeed be proven to be incomplete in some way, but it would be folly to expect that they'd be found fundamentally wrong after withstanding such rigorous testing.

Nanobanano
1 / 5 (1) Sep 13, 2011
It's all a scam by wine and cheese eating scientists who are out to steal my money.

These cushy scientists jobs and their porshes and disgusting lifestyles with interns and college children must be stopped.

Defund science now. All of it. It's the Devil's work.


Who are you mocking? not me, I hope.

Defunding science is far from anything I'd support.

I think you are referring to the "conservative" Christians, who typically prefer to remain as ignorant and uneducated as possible regarding math and science, and are content to take instruction from a "pastor" who was, as it turns out, a high school drop out. Then they spend all day long arguing with other pastors about doctrine all day long over stupid one word discrepencies, and they know nothing about the real world? Sound familliar?

I know a few who literally never read any book before in their life except a Bible. It's disgraceful.

It also sounds like Republicans too.
antialias_physorg
5 / 5 (1) Sep 14, 2011
It's not error-proof. It's self-correcting. Here is but the latest in a long, *long* example of such cases where what was previously thought correct was rewritten or refined.


Yes.

Add to that that doing a correct statistical analysis is extremely hard - and most scientists are not experts in that field (but rather in their field of study). I know. I had to plough through additional statistics courses just to understand whether the analyses I was doing were using the correct statistical methods:

- you have to choose a test for the data
- you have to test the data whether it even confoms to a specific distribution
- you have to test whether the *metric* for testing for a specific distribution is applicable to your data
- ...

There's such an abundance of methods and the differences are so esoteric that it boggles the mind - and I can readily see where people can erroneously choose an inappropriate measure (and how reviewers can miss that)

Callippo
1 / 5 (1) Sep 14, 2011
Science is self-correcting. It's not error-proof. It's self-correcting.

But the money are already spent...;-)

Apparently, more stringent public feedback would work better, than the relying on self-correcting mechanism from inside. For example, we can say, the communism at Cuba or North Korea is self-correcting too. Even East Asia terrorists are self-correcting occasionally. But it's more effective to intervene against them from outside, because their system lack self-reflection.

The memo simply is, if you get the money to large group of people under the hope, they will deal with them effectively, the communism will always remain: with all its wastes, censorship and ineffectiveness. It's easy to predict result - no matter how these people are honest individually.
Callippo
not rated yet Sep 14, 2011
"if you get the money" should be "if you give the money"
Truthforall
not rated yet Sep 14, 2011
This might not be an intentional fault since Neuro-scientists who are expert in their own fields are, very often than not, less skilled in statistics.
The journals as gate keepers on the other hand should have the resource and experience to include prominent mathematicians as reviewers to scrutinize the sampling method and statistical analysis instead of passing the papers only to other scientists in the same field.
This then becomes pal review not critical peer review. Should the reviewers happen to employ or inherit the same mistake in their own work, this genetic defect will pass on. When new researches cite and build on these defective papers, the damage in terms of wasted effort can be phenominal.
Now that we know of this problem, what are we going to do about it?
tjcoop3
not rated yet Sep 14, 2011

Now that we know of this problem, what are we going to do about it?

My money is on "Not one blasted thing". The system does not and will not correct itself. There is too much pride and arrogance, let alone monies and reputations involved. Journals often exclude the best research because it contradicts the status quo. Nothing controversial or new ya know.
With Big Pharma and Big Medicine involved in so much here and the tendency to prefer treatment to cure in current practice there is little incentive to change anything about the way research is done.
Unless you have a trauma wound or a broken bone you are statistically more likely to live longer by researching cures over the internet than going to your physician.
Vendicar_Decarian
3 / 5 (2) Sep 14, 2011
"There is too much pride and arrogance, let alone monies and reputations involved. Journals often exclude the best research because it contradicts the status quo." - tjCoop

That is why the reality of Neutron/Neutron repulsion as the power source for the neutron star at the center of the sun is being suppressed.

There is just too much money being spent on the study of stars that astronomers claim are made from hydrogen but in fact can't see.

Solar astronomers will never advance until they realize that they can not see the sun.
Vendicar_Decarian
3 / 5 (2) Sep 14, 2011
"Unless you have a trauma wound or a broken bone you are statistically more likely to live longer by researching cures over the internet than going to your physician." - TJCoop

Darn right. Lord Moncton of Betchley - famous climate change denialist - has a salve that cures cancer, menopause, hair loss, testicular lesions and AIDS. among other things.

He makes it in his basement I think, from ground bull testicles , milk of magnesia, and Plowtuneium Macrofloxide.
jdbertron
1 / 5 (1) Sep 14, 2011
Here is a post of the paper:
http://www.sander...tive.pdf
The abstract and the post are both meaningless. "they cannot then, reasonably compare the two results and come up with something that they consider significant ". You can always compare results and make hypotheses. Drawing conclusions is what can get you in trouble.
Did they mean compare the "groups" to draw a conclusion ?
antialias_physorg
5 / 5 (1) Sep 15, 2011
The journals as gate keepers on the other hand should have the resource and experience to include prominent mathematicians as reviewers to scrutinize the sampling method and statistical analysis instead of passing the papers only to other scientists in the same field.

The journals just hand the papers they receive out to the peer reviewers. These are other researchers in the field and - incidentally - don't get paid a dime for working through the paper. It is just considered good manners in scientific circles to review papers when asked.

(Also there's the slight benefit of getting to be one of the first to look at new results - but since everyone knows everyone else in those fields and research times are on the order of years that 'advantage' is very small. But it's still cool to be one of the first to know the exact results of the latest resaerch.)

Journals have jouranlists and editors on hand - not statisticians.