Research team quantifies 'the difficulties of reproducibility'

Nov 28, 2013

(Phys.org) —A key pillar of "the scientific method" is reproducibility, one way to prove another scientist's experimental claims. If the experiment and its results can be reproduced, the validity of the work is considerably strengthened.

But scientific reproducibility is not as common or as easy as many non-scientists think. In a recent study of landmark papers in cancer research, for example, only 11 percent of the studies could be reproduced.

In another recent case, a graduate student failed to reproduce the results of a widely cited economic-policy paper – a failure which led to the exposure of significant, but unintentional, errors.

Hoping to quantify just what it takes to reproduce a scientific paper, researchers from three institutions conducted a study of a computational biology paper that analyzed tuberculosis-drug targets.

Philip Bourne, professor of pharmacology at the Skaggs School of Pharmacy and Pharmaceutical Sciences at the University of California San Diego, the principal investigator of the tuberculosis study and co-author of the paper; Daniel Garijo, a doctoral student from the Universidad Politecnica of Madrid; and Yolanda Gil, professor of computer science at the University of Southern California, collaborated to quantify "the difficulties of reproducibility" – and to suggest a possible solution.

Writing in the journal PLOS ONE, Gil and Garijo reported that they had to spend "significant time" reviewing materials from Bourne's lab, and talking to previous lab members, to satisfactorily reconstruct the computational experiments of the original paper.

"We estimated the overall time to reproduce the method at 280 hours for a novice with minimal expertise in bioinformatics," said Garijo, "either because computer scripts were not available, or there were assumptions in the described methods that would not be obvious to a non-expert."

Failure to reproduce a study is rarely the result of fraud, said Bourne, but "mostly lack of a complete record." In this case, he said, "it was not that the work could not be reproduced; the problem was that it took so much time – something all new graduate students in the lab can verify as they pick up previous students' work."

In this day and age, said Bourne, "We should really be doing better. It's unfortunate to say this about my own work – but how many scientists could claim to be doing better?"

One way scientists might do better, said Gil, is to do what she and Garijo did. "As part of the reconstructive work," she said, "we encoded the computational experiment in a semantic workflow, shared as a web object with annotations of its meanings."

These workflow systems are now reaching such a level of maturity, say the researchers, that they're likely to be adopted more broadly. "This should greatly facilitate reproducibility," their report asserts.

Journals and their publishers can also encourage improved reproducibility by insisting that workflows, data, and software to be part of the submission-and-review process, the authors say.

Finally, they note, better reproducibility may eventually be mandated, citing a recent administration memorandum asking all agencies to develop policies to make results of all federally funded research broadly available to scientists, industry, and the public.

Explore further: Scientific reproducibility is hampered by a lack of specificity of the material resources

Related Stories

New computational technique can predict drug side effects

Dec 11, 2007

Early identification of adverse effects of drugs before they are tested in humans is crucial in developing new therapeutics, as unexpected effects account for a third of all drug failures during the development process.

Recommended for you

Not just the poor live hand-to-mouth

5 hours ago

When the economy hits the skids, government stimulus checks to the poor sometimes follow. Stimulus programs—such as those in 2001, 2008 and 2009—are designed to boost the economy quickly by getting cash ...

Math modeling handbook now available

7 hours ago

Math comes in handy for answering questions about a variety of topics, from calculating the cost-effectiveness of fuel sources and determining the best regions to build high-speed rail to predicting the spread ...

Archaeologists, tribe clash over Native remains

8 hours ago

Archaeologists and Native Americans are clashing over Indian remains and artifacts that were excavated during a construction project in the San Francisco Bay Area, but then reburied at an undisclosed location.

Male-biased tweeting

10 hours ago

Today women take an active part in public life. Without a doubt, they also converse with other women. In fact, they even talk to each other about other things besides men. As banal as it sounds, this is far ...

Developing nations ride a motorcycle boom

11 hours ago

Asia's rapidly developing economies should prepare for a full-throttle increase in motorcycle numbers as average incomes increase, a new study from The Australian National University has found.

User comments : 1

Adjust slider to filter visible comments by rank

Display comments: newest first

julianpenrod
1 / 5 (5) Nov 28, 2013
Very interesting that this same day, PhysOrg has another article on a test, evidently using unreliable anonymous information from anyone with a computer, that "reproduces" results in "psychology" that formerly were irreproducible. "Science" may be close to having it exposed that it's all hogwash. There, as here, "scientists" doing swindles in conventional areas have their frauds excused for "mistakes" rather than criminality on their part. When unconventional studies get reproducible results don't expect there necessarily won't be a "scientist" who says the reproducibility proves it's a lie! Why, for example, the issue of the scripts for the tuberculosis "experiment" not being available? Did they use a different definition for the mean or the standard deviation? All "science" is a scam.

More news stories

Not just the poor live hand-to-mouth

When the economy hits the skids, government stimulus checks to the poor sometimes follow. Stimulus programs—such as those in 2001, 2008 and 2009—are designed to boost the economy quickly by getting cash ...

Male-biased tweeting

Today women take an active part in public life. Without a doubt, they also converse with other women. In fact, they even talk to each other about other things besides men. As banal as it sounds, this is far ...

Archaeologists, tribe clash over Native remains

Archaeologists and Native Americans are clashing over Indian remains and artifacts that were excavated during a construction project in the San Francisco Bay Area, but then reburied at an undisclosed location.

Math modeling handbook now available

Math comes in handy for answering questions about a variety of topics, from calculating the cost-effectiveness of fuel sources and determining the best regions to build high-speed rail to predicting the spread ...