August 29, 2014 report
Researchers suggest lack of published null result papers skews reliability of those that are published
(Phys.org) —A trio of researchers at Stanford University has shined a light on a problem many in the social science research arena are aware of but tend to ignore: the problem of null result papers not being written or published. In their paper published in the journal Science, Annie Franco, Neil Malhotra and Gabor Simonovits suggest that not publishing null result papers produces a bias in the literature, skewing the reliability of papers with strong results that are published. Jeffery Mervis offers an In Depth piece on the team's work in the same journal edition.
What should a social scientist do if he or she comes up with a hypotheses regarding human behavior, designs a way to test it, runs that test, but then learns that nothing new has been found by doing so? At first blush, it might seem logical to toss the idea into the trash, or file drawer and move on to something else, which is exactly what a lot of researchers do, the researchers with this new effort report. After all, if you don't find anything relevant or pertinent, others might think you didn't actually accomplish anything, so why would you write up a paper describing what happened and submit it to a journal?
The answer lies in the domain of published results, if respected journals only ever publish strong result papers, an impression is created that only research that provides strong results is important, which of course is nonsense. It also leaves the science open to wasted effort when other researchers come up with the same hypotheses and the same result.
To learn more about the problem, the researchers pulled data from TESS, an online program that allows researchers to get data from surveys that have been conducted as part of research efforts sponsored by the National Science Foundation. The team found that only 48 percent of studies begun were completed, so they contacted the study leaders to find out what happened to those that weren't represented. Their work revealed that just 20 percent of null result papers wound up being published, and that an astounding 65 percent of the null result studies had even resulted in a written paper—the researchers had simply walked away. When asked why, many suggested that to do so would be wasted effort as there would be little interest by journals.
The researchers suggest that perhaps a new repository be set up for the placement of null result papers, one that would be accessible by other researchers. That would allow for a future scenario when a scientist could ask their computer about an idea, and get back a history of the research surrounding it, rather than a skewed list that shows only the work of successful endeavors.
We study publication bias in the social sciences by analyzing a known population of conducted studies—221 in total—where there is a full accounting of what is published and unpublished. We leverage TESS, an NSF-sponsored program where researchers propose survey-based experiments to be run on representative samples of American adults. Because TESS proposals undergo rigorous peer review, the studies in the sample all exceed a substantial quality threshold. Strong results are 40 percentage points more likely to be published than null results, and 60 percentage points more likely to be written up. We provide not only direct evidence of publication bias, but also identify the stage of research production at which publication bias occurs—authors do not write up and submit null findings.
© 2014 Phys.org