Study finds statistical error in large numbers of neuroscience papers
September 13, 2011 by Bob Yirka 
(PhysOrg.com) -- Sander Nieuwenhuis and his associates from the Netherlands have done a study on one particular type of statistical error that apparently crops up in an inordinately large number of papers published in neuroscience journals. In their paper, published in Nature Neuroscience, they claim that up to half of all papers published in such journals contain the error.
The problem lies in the way findings are presented. If a group of researchers, for example, applies something (chemical, food, energy, etc.) that is to cause an effect on something else (nerve cells, populations, inanimate objects, etc.) and finds the amount of change caused by the main thing that is being studied is “significant” but the change in the control group is not, they cannot then, reasonably compare the two results and come up with something that they consider significant unless the differences between the two are actually statistically significant (based on additional research).
Why such errors appear in so many research papers is open to debate. Whether it’s due to researchers wishing to overstate their findings, ignorance, or simple sloppiness, it’s clear that more scrutiny and peer review must be done by researchers before submitting their work. Of course, that’s only half the equation, why are journals who obviously take their reputations very seriously not properly vetting such papers before publishing them?
In their study, the group reviewed 513 papers published in five different highly regarded journals over a two year period. They found half of the papers (where such an error was possible) had the error in them. In addition they also found that when looking at 120 articles published on Nature Neuroscience (with cell and molecular themes) that 25 had the error in them.
Clearly there is a serious problem here; this research project highlights a problem that is likely present in other areas of science as well; namely the inaccuracies present in science journals, mainstream science magazines, the media and perhaps even in classroom lectures. Failing to check for and fix simple statistical inaccuracies in papers presenting results obtained in research, calls into question their very integrity.
Hopefully research studies such as this one will cause alarm both in the research and publishing communities and bring about better controls on both.
More information: Erroneous analyses of interactions in neuroscience: a problem of significance, Nature Neuroscience 14, 1105–1107 (2011) doi:10.1038/nn.2886
Abstract
In theory, a comparison of two experimental effects requires a statistical test on their difference. In practice, this comparison is often based on an incorrect procedure involving two separate tests in which researchers conclude that effects differ when one effect is significant (P < 0.05) but the other is not (P > 0.05). We reviewed 513 behavioral, systems and cognitive neuroscience articles in five top-ranking journals (Science, Nature, Nature Neuroscience, Neuron and The Journal of Neuroscience) and found that 78 used the correct procedure and 79 used the incorrect procedure. An additional analysis suggests that incorrect analyses of interactions are even more common in cellular and molecular neuroscience. We discuss scenarios in which the erroneous procedure is particularly beguiling.
via Guardian
© 2011 PhysOrg.com
-
From lemons to lemonade: Reaction uses carbon dioxide to make carbon-based semiconductor,
32 comments
-
Thioridazine kills cancer stem cells in human while avoiding toxic side-effects of conventional cancer treatments,
3 comments
-
SpaceX private rocket blasts off for space station (Update),
42 comments
-
Climate scientists say they have solved riddle of rising sea,
31 comments
-
SpaceX capsule has 'new car' smell, astronauts say (Update),
4 comments
-
Justifying Proof by Contradiction
7 hours ago
-
Combining equations help
8 hours ago
-
About the definition of "discrete random variable"
10 hours ago
-
Limits
May 26, 2012
-
Complex numbers: Why is the modulus of z...
May 26, 2012
-
A close approximation for square root of 2.
May 25, 2012
- More from Physics Forums - General Math
More news stories
Change in developmental timing was crucial in the evolutionary shift from dinosaurs to birds: study
At first glance, it's hard to see how a common house sparrow and a Tyrannosaurus Rex might have anything in common. After all, one is a bird that weighs less than an ounce, and the other is a dinosaur that ...
Other Sciences / Archaeology & Fossils
4 hours ago |
5 / 5 (2) |
0
|
Social welfare cuts ultimately come with heavy price, researchers say
(Phys.org) -- Slashing government funding for Medicaid, food stamps and other programs that serve the poor – while politically popular with some lawmakers and many conservatives – may do more harm ...
Other Sciences / Social Sciences
May 24, 2012 |
4.3 / 5 (16) |
154
Ancient Bethlehem seal unearthed in Jerusalem
Israeli archaeologists have discovered a 2,700-year-old seal that bears the inscription "Bethlehem," the Israel Antiquities Authority announced Wednesday, in what experts believe to be the oldest artifact ...
Other Sciences / Archaeology & Fossils
May 23, 2012 |
3.5 / 5 (14) |
24
Dollars and sense: Why are some people morally against tax?
As the U.S. presidential election campaigns heat up, the economic debate is dominated by bailouts, austerity and, inevitably, taxation. Now a new study published in Symbolic Interaction asks why tax is such an important issue ...
Other Sciences / Social Sciences
May 23, 2012 |
3 / 5 (2) |
15
Oldest Jewish archaeological evidence on the Iberian Peninsula
German archaeologists of the Friedrich Schiller University Jena found one of the oldest archaeological evidence so far of Jewish Culture on the Iberian Peninsula at an excavation site in the south of Portugal, ...
Other Sciences / Archaeology & Fossils
May 25, 2012 |
4.2 / 5 (6) |
12
Nvidia trumpets Tegra 3 phone design wins for 2012
(Phys.org) -- Nvidia’s competitive war paint has a name, Tegra 3. On the heels of Nvidia announcements about lowering costs of its Tegra 3 processors and Nvidia-enabled tablets running Android Ice Cream ...
'Unzipped' carbon nanotubes could help energize fuel cells, batteries
Multi-walled carbon nanotubes riddled with defects and impurities on the outside could replace some of the expensive platinum catalysts used in fuel cells and metal-air batteries, according to scientists at ...
T cells 'hunt' parasites like animal predators seek prey, study shows
By pairing an intimate knowledge of immune-system function with a deep understanding of statistical physics, a cross-disciplinary team at the University of Pennsylvania has arrived at a surprising finding: T cells use a movement ...
Computer model used to pinpoint prime materials for efficient carbon capture
When power plants begin capturing their carbon emissions to reduce greenhouse gases – and to most in the electric power industry, it's a question of when, not if – it will be an expensive undertaking.
Land and sea species differ in climate change response: study
(Phys.org) -- Marine and terrestrial species will likely differ in their responses to climate warming, new research by Simon Fraser University and Australia’s University of Tasmania has found.
Scientist: Evolution debate will soon be history
(AP) -- Richard Leakey predicts skepticism over evolution will soon be history. Not that the avowed atheist has any doubts himself.
Sep 13, 2011
Rank: 2 / 5 (4)
Got so bad they write peer reviewed papers on the fallacies and inaccuracies in other peer reviewed papers, has it?
Sep 13, 2011
Rank: 2.3 / 5 (9)
The conspiracy among these corrupt Neuro-scientists is being exposed, and their reign of socialist terror is coming to an end.
Sep 13, 2011
Rank: 5 / 5 (1)
Is this real? Or is this a Poe?
Sep 13, 2011
Rank: 5 / 5 (1)
Ha ha I thought the exact same thing. I hope this person is kidding. I believe there is a chance that human cognition may lie outside (or separately) from the brain itself, but that "theory" has nothing to do with the bible or a silly god.
Sep 13, 2011
Rank: 3 / 5 (6)
These cushy scientists jobs and their porshes and disgusting lifestyles with interns and college children must be stopped.
Defund science now. All of it. It's the Devil's work.
Sep 13, 2011
Rank: 5 / 5 (3)
It's not error-proof. It's self-correcting. Here is but the latest in a long, *long* example of such cases where what was previously thought correct was rewritten or refined.
What many want to take from that is that we cannot trust anything Science tells us. That's unfortunate, as a great deal of 'mature' science has withstood uncountable attempts at proving it wrong or otherwise incomplete. Those cases may indeed be proven to be incomplete in some way, but it would be folly to expect that they'd be found fundamentally wrong after withstanding such rigorous testing.
Sep 13, 2011
Rank: 1 / 5 (1)
Who are you mocking? not me, I hope.
Defunding science is far from anything I'd support.
I think you are referring to the "conservative" Christians, who typically prefer to remain as ignorant and uneducated as possible regarding math and science, and are content to take instruction from a "pastor" who was, as it turns out, a high school drop out. Then they spend all day long arguing with other pastors about doctrine all day long over stupid one word discrepencies, and they know nothing about the real world? Sound familliar?
I know a few who literally never read any book before in their life except a Bible. It's disgraceful.
It also sounds like Republicans too.
Sep 14, 2011
Rank: 5 / 5 (1)
Yes.
Add to that that doing a correct statistical analysis is extremely hard - and most scientists are not experts in that field (but rather in their field of study). I know. I had to plough through additional statistics courses just to understand whether the analyses I was doing were using the correct statistical methods:
- you have to choose a test for the data
- you have to test the data whether it even confoms to a specific distribution
- you have to test whether the *metric* for testing for a specific distribution is applicable to your data
- ...
There's such an abundance of methods and the differences are so esoteric that it boggles the mind - and I can readily see where people can erroneously choose an inappropriate measure (and how reviewers can miss that)
Sep 14, 2011
Rank: 1 / 5 (1)
But the money are already spent...;-)
Apparently, more stringent public feedback would work better, than the relying on self-correcting mechanism from inside. For example, we can say, the communism at Cuba or North Korea is self-correcting too. Even East Asia terrorists are self-correcting occasionally. But it's more effective to intervene against them from outside, because their system lack self-reflection.
The memo simply is, if you get the money to large group of people under the hope, they will deal with them effectively, the communism will always remain: with all its wastes, censorship and ineffectiveness. It's easy to predict result - no matter how these people are honest individually.
Sep 14, 2011
Rank: not rated yet
Sep 14, 2011
Rank: not rated yet
The journals as gate keepers on the other hand should have the resource and experience to include prominent mathematicians as reviewers to scrutinize the sampling method and statistical analysis instead of passing the papers only to other scientists in the same field.
This then becomes pal review not critical peer review. Should the reviewers happen to employ or inherit the same mistake in their own work, this genetic defect will pass on. When new researches cite and build on these defective papers, the damage in terms of wasted effort can be phenominal.
Now that we know of this problem, what are we going to do about it?
Sep 14, 2011
Rank: not rated yet
My money is on "Not one blasted thing". The system does not and will not correct itself. There is too much pride and arrogance, let alone monies and reputations involved. Journals often exclude the best research because it contradicts the status quo. Nothing controversial or new ya know.
With Big Pharma and Big Medicine involved in so much here and the tendency to prefer treatment to cure in current practice there is little incentive to change anything about the way research is done.
Unless you have a trauma wound or a broken bone you are statistically more likely to live longer by researching cures over the internet than going to your physician.
Sep 14, 2011
Rank: 3 / 5 (2)
That is why the reality of Neutron/Neutron repulsion as the power source for the neutron star at the center of the sun is being suppressed.
There is just too much money being spent on the study of stars that astronomers claim are made from hydrogen but in fact can't see.
Solar astronomers will never advance until they realize that they can not see the sun.
Sep 14, 2011
Rank: 3 / 5 (2)
Darn right. Lord Moncton of Betchley - famous climate change denialist - has a salve that cures cancer, menopause, hair loss, testicular lesions and AIDS. among other things.
He makes it in his basement I think, from ground bull testicles , milk of magnesia, and Plowtuneium Macrofloxide.
Sep 14, 2011
Rank: not rated yet
http://www.sander...tive.pdf
The abstract and the post are both meaningless. "they cannot then, reasonably compare the two results and come up with something that they consider significant ". You can always compare results and make hypotheses. Drawing conclusions is what can get you in trouble.
Did they mean compare the "groups" to draw a conclusion ?
Sep 15, 2011
Rank: 5 / 5 (1)
The journals just hand the papers they receive out to the peer reviewers. These are other researchers in the field and - incidentally - don't get paid a dime for working through the paper. It is just considered good manners in scientific circles to review papers when asked.
(Also there's the slight benefit of getting to be one of the first to look at new results - but since everyone knows everyone else in those fields and research times are on the order of years that 'advantage' is very small. But it's still cool to be one of the first to know the exact results of the latest resaerch.)
Journals have jouranlists and editors on hand - not statisticians.