Scientists unknowingly tweak experiments

March 18, 2015
Dr Megan Head in her evolutionary biology lab at the Research School of Biology. Credit: Regina Vega-Trejo

A new study has found some scientists are unknowingly tweaking experiments and analysis methods to increase their chances of getting results that are easily published.

The study conducted by ANU scientists is the most comprehensive investigation into a type of called p-hacking.

P-hacking happens when researchers either consciously or unconsciously analyse their data multiple times or in multiple ways until they get a desired result. If p-hacking is common, the exaggerated results could lead to misleading conclusions, even when evidence comes from multiple studies.

"We found evidence that p-hacking is happening throughout the life sciences," said lead author Dr Megan Head from the ANU Research School of Biology.

The study used text mining to extract p-values - a number that indicates how likely it is that a result occurs by chance - from more than 100,000 research papers published around the world, spanning many scientific disciplines, including medicine, biology and psychology.

"Many researchers are not aware that certain methods could make some results seem more important than they are. They are just genuinely excited about finding something new and interesting," Dr Head said.

"I think that pressure to publish is one factor driving this bias. As scientists we are judged by how many publications we have and the quality of the they go in.

"Journals, especially the top journals, are more likely to publish experiments with new, interesting results, creating incentive to produce results on demand."

Dr Head said the study found a high number of p-values that were only just over the traditional threshold that most scientists call statistically significant.

"This suggests that some scientists adjust their experimental design, datasets or statistical methods until they get a result that crosses the significance threshold," she said.

"They might look at their results before an experiment is finished, or explore their data with lots of different , without realising that this can lead to bias."

The concern with p-hacking is that it could get in the way of forming accurate scientific conclusions, even when scientists review the evidence by combining results from multiple studies.

For example, if some studies show a particular drug is effective in treating hypertension, but other studies find it is not effective, would analyse all the data to reach an overall conclusion. But if enough results have been p-hacked, the drug would look more effective than it is.

"We looked at the likelihood of this bias occurring in our own specialty, evolutionary biology, and although p-hacking was happening it wasn't common enough to drastically alter general that could be made from the research," she said.

"But greater awareness of p-hacking and its dangers is important because the implications of p-hacking may be different depending on the question you are asking."

The research is published in PLOS Biology.

Explore further: Bias pervades the scientific reporting of animal studies

More information: "The Extent and Consequences of P-Hacking in Science." PLOS Biology, DOI: 10.1371/journal.pbio.1002106

Related Stories

Bias pervades the scientific reporting of animal studies

July 17, 2013

A new study published in the open access journal PLOS Biology suggests that the scientific literature could be compromised by substantial bias in the reporting of animal studies, and may be giving a misleading picture of ...

Feds say third guilty plea entered in computer hacking case

January 21, 2015

Federal prosecutors have obtained a third guilty plea involving an international hacking ring that gained access to a U.S. Army computer network while targeting computer giant Microsoft and several video game developers.

Recommended for you

An inflexible diet led to the disappearance of the cave bear

August 23, 2016

Senckenberg scientists have studied the feeding habits of the extinct cave bear. Based on the isotope composition in the collagen of the bears' bones, they were able to show that the large mammals subsisted on a purely vegan ...

28 comments

Adjust slider to filter visible comments by rank

Display comments: newest first

cantdrive85
3.3 / 5 (3) Mar 18, 2015
"Unknowingly"....
SamB
2.3 / 5 (9) Mar 18, 2015
Sounds just like the way the Church of Global Warming massages their data!
Scroofinator
3 / 5 (2) Mar 18, 2015
Working the data until it looks the way they want it and not necessarily showing how it really is.

The scientific method at its finest...
antialias_physorg
5 / 5 (1) Mar 18, 2015
OK, I can see where p-hacking would be something that leads to a false conclusion. But reviewers should catch that.
How to select the right statistical methods is something that every scientists needs to learn. But even if you know how to do this the devil is often in the details of the choice.

Choosing the 'wrong' method will not give you huge discrepancies in the results but can be the thing that shifts it from barely not significant to statistically significant.

That said I find it not very surprising that there is clustering of p-values in papers near (and slightly below) the statistically significant cutoff.
1) Easy stuff has already been researched. Science happens on the edge. Statistical significance defines that edge.
2) p-values above the statistical significant cutoff point don't get published (or at the very least the p-value is not cited in the publication. There is no point in publishing it beyond saying: "didn't work" ).
MR166
5 / 5 (2) Mar 18, 2015
"P-hacking happens when researchers either consciously or unconsciously analyse their data multiple times or in multiple ways until they get a desired result."

Hummm, now why would there ever be a "Desired Result"? Could government funding have anything to do with that. I have been told that this idea is ludicrous since all scientists are pure and noble.
indio007
5 / 5 (2) Mar 18, 2015
This is why the scientific method is
Experiment -> theory
not
Theory->experiment
Dethe
2.3 / 5 (3) Mar 18, 2015
This is why the scientific method is Experiment -> theory not Theory->experiment
It can be both, just in balanced way. Most of experiments are planned for falsification of existing theories, which is the basis of Popper methodology of science.
Whydening Gyre
5 / 5 (3) Mar 18, 2015
Kinda makes me wonder if THIS study was subject to the same p-hack factor...
MR166
5 / 5 (2) Mar 18, 2015
Most likely you are correct WG. It is very hard to be totally unbiased as a human being.
Dethe
2.3 / 5 (3) Mar 18, 2015
P-hacking has its counterpart in d-hacking, i.e. in systematical ignorance or even dismissal of findings, which don't fit the intersubjective belief of given era. Without it we wouldn't have megawatt energy factories running on phenomena, which officially doesn't exist according to mainstream science. The d-hacking usually manifest itself with unjustifiable delay between finding and peer-reviewed attempt for verification of this finding.
MR166
5 / 5 (1) Mar 18, 2015
"P-hacking has its counterpart in d-hacking, i.e. in systematical ignorance or even dismissal of findings, which don't fit the intersubjective belief of given era."

Every generation had their stories about "scientific" breakthroughs involving free power. Mine had the pills that you dropped into you gas tank that turned plain tap water into gasoline.

You have Rossi and cold fusion.
Dethe
1 / 5 (1) Mar 18, 2015
This is nice and all, but the hydrogen fusion at nickel was never attempted to replicate in peer-review study, thus indicating the d-hacking of reality. We should do experiments first, just after then to judge it - this is how the science should be done.
MR166
5 / 5 (3) Mar 18, 2015
"We should do experiments first, just after then to judge it - this is how the science should be done."

I agree with you 101%, just don't try to tell me that this is a proven product and that it is a biased populace that is keeping it from being produced.
Dethe
1 / 5 (2) Mar 18, 2015
Without your own experiments you cannot tell anything about E-Cat. The only scientist here is A. Rossi - he has a product, he presented public demo with scientists twice-times, has sold it and the test run at customer proceeds well. In addition, we have additional replications of his system already (Parkhomov, Aherm and others). The only trolls falsifying the reality here are the mainstream physicists, who did absolutely nothing in this matter during last twenty years.
MR166
5 / 5 (1) Mar 18, 2015
Dethe if Rossi's product really does work there will be 100s of Billions of dollars available to him from private investors. That will be the final undeniable proof of who is correct. Just don't try to tell me that some coal/energy company somewhere is trying to interfere with it's development and implementation.
Dethe
1 / 5 (2) Mar 18, 2015
there will be 100s of Billions of dollars available to him from private investors
Why it should be, if just the most wealthy investors are motivated to delay in acceptation of cold fusion in the name of protection of their previous investments? This is also the reason, why nobody does invest into production of domestic E-Cat units, which would threat the existing investments even more.
That will be the final undeniable proof of who is correct
The science has its own criterions of validity of experimental findings, but it doesn't use them. Or better to say, it does use them only if it brings some advantage for scientific community. The interests of society which is actually paying the research are heartily ignored during it. In another words, the scientists created a silent lobby for our own money, which is separated from the needs of human society. The more money they will get, the more they will be hostile toward interests of human civilization as a whole.
Dethe
1 / 5 (2) Mar 18, 2015
The d-hacking is not just about cold fusion (where it is just a most apparent) - the same situation exists in many other areas - the Big Pharma or superconductivity/antigravity research is working on the same principles (1, 2, 3). It's a widespread problem, which really delays the future progress of human civilization.
jeffensley
1 / 5 (2) Mar 18, 2015
Very glad to see science checking itself. This seems to fit most modern "modelling". Reminds of a guy I know who claims he can "witch" where underground lines are buried. I notice him subtly twisting his wrist to make the rods turn but I wouldn't be surprised if he wasn't consciously aware of it. He has an idea in his head of where lines are running before he even starts and he subconsciously "verifies" it with his rods.
jeffensley
1 / 5 (2) Mar 18, 2015
This is why the scientific method is
Experiment -> theory
not
Theory->experiment


That's how it SHOULD be but one gets the sense that only the latter brings in research grants and thus is how most science works today.
gkam
3 / 5 (2) Mar 18, 2015
This is why reputable science is Peer-Reviewed!
Whydening Gyre
not rated yet Mar 18, 2015
Reminds of a guy I know who claims he can "witch" where underground lines are buried. I notice him subtly twisting his wrist to make the rods turn but I wouldn't be surprised if he wasn't consciously aware of it. He has an idea in his head of where lines are running before he even starts and he subconsciously "verifies" it with his rods.

Not if the rods are just an antennae relaying info his own internal "electricity radar", which CAUSES the reaction...:-)
Rydog
5 / 5 (1) Mar 18, 2015
Scientists are changing their experiments? It's almost like they're trying to find something new and publishable. Who told them they could do that?
Mishery
5 / 5 (1) Mar 19, 2015
I used to work in cognitive psychology and p hacking is the norm. It was one reason I left. I didn't know what it was called then, but it was clearly wrong but no one seemed to care.
MR166
5 / 5 (1) Mar 19, 2015
"Reminds of a guy I know who claims he can "witch" where underground lines are buried."

WG I personally have done that and it works. My brother in law worked for the water department and showed me how to do that with two bent steel rods about the diameter of a close hanger. Underground pipes, even non metallic ones can be located that way. If you doubt me just ask any old timer at your local public works dept. You can actually see the tips of the rods move towards each other as you pass over a pipe.
antialias_physorg
5 / 5 (3) Mar 19, 2015
Couple of years back they did an investigative bit of journalism and got the top self-professed water-witchers together and had them walk over 10 pipes, one of which was randomly filled with running water. The witchers designed the test themselves (depth of burying, water pressure, diameter of pipes) so it would be fair. They were each confident of getting etween 40% and 90% correct.

Needless to say: the results, after many trials, was well within the statistical region of "just guessing" (They got 11% right (cumulative)...and none of them was individually above 15% IIRC).
MR166
3 / 5 (2) Mar 19, 2015
" had them walk over 10 pipes, one of which was randomly filled with running water"

Anti if the criteria in passing the test was to find which pipe had the water I can see why they failed. I am claiming that this method will find the pipe itself irregardless of whether or not there is water in it. I think that the method is just detecting a void in the ground. I was using steel welding rod so perhaps some sort of magnetic distortion was created by the void.
TyroneTork
5 / 5 (2) Mar 21, 2015
The good news is that if one believes the study, then one would assume that the p-hacking data itself was p-hacked to get published, so therefore overestimating overall p-hacking.
Although this of course means that there is slightly less chance the study itself was p-hacked, I'm pretty sure, after running a few calculations myself, that it is really a big deal in some way!
nevermark
5 / 5 (2) Mar 22, 2015
The fact that everyone has unrecognized biases is EXACTLY WHY SCIENCE WAS INVENTED!

Peer review, public publishing, falsifiable tests, reproducibility: all these self-correcting mechanisms invented precisely because everyone, even the most honest scientists, have some bias or can make honest (or not so honest) mistakes.

Most scientists actively try to eliminate their own bias, which is always the best first step and makes it likely that their results will stand the test of time. But even the best intentioned by overlook a fudge here or an optimistic interpretation there.

What the cynics of science don't get, is that the scientific process minimizes and weeds out the effects of biases over time. Instead, the cynics say "science isn't perfect" therefore "my pet theory is being suppressed" or "this validates my denial of what I don't want to believe".

The irony is that is extremely biased thinking without any of the self-correcting mechanisms that science brings to bear.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.