Research team quantifies 'the difficulties of reproducibility'

November 28, 2013

( —A key pillar of "the scientific method" is reproducibility, one way to prove another scientist's experimental claims. If the experiment and its results can be reproduced, the validity of the work is considerably strengthened.

But scientific reproducibility is not as common or as easy as many non-scientists think. In a recent study of landmark papers in cancer research, for example, only 11 percent of the studies could be reproduced.

In another recent case, a graduate student failed to reproduce the results of a widely cited economic-policy paper – a failure which led to the exposure of significant, but unintentional, errors.

Hoping to quantify just what it takes to reproduce a scientific paper, researchers from three institutions conducted a study of a computational biology paper that analyzed tuberculosis-drug targets.

Philip Bourne, professor of pharmacology at the Skaggs School of Pharmacy and Pharmaceutical Sciences at the University of California San Diego, the principal investigator of the tuberculosis study and co-author of the paper; Daniel Garijo, a doctoral student from the Universidad Politecnica of Madrid; and Yolanda Gil, professor of computer science at the University of Southern California, collaborated to quantify "the difficulties of reproducibility" – and to suggest a possible solution.

Writing in the journal PLOS ONE, Gil and Garijo reported that they had to spend "significant time" reviewing materials from Bourne's lab, and talking to previous lab members, to satisfactorily reconstruct the computational experiments of the original paper.

"We estimated the overall time to reproduce the method at 280 hours for a novice with minimal expertise in bioinformatics," said Garijo, "either because computer scripts were not available, or there were assumptions in the described methods that would not be obvious to a non-expert."

Failure to reproduce a study is rarely the result of fraud, said Bourne, but "mostly lack of a complete record." In this case, he said, "it was not that the work could not be reproduced; the problem was that it took so much time – something all new graduate students in the lab can verify as they pick up previous students' work."

In this day and age, said Bourne, "We should really be doing better. It's unfortunate to say this about my own work – but how many scientists could claim to be doing better?"

One way scientists might do better, said Gil, is to do what she and Garijo did. "As part of the reconstructive work," she said, "we encoded the computational experiment in a semantic workflow, shared as a web object with annotations of its meanings."

These workflow systems are now reaching such a level of maturity, say the researchers, that they're likely to be adopted more broadly. "This should greatly facilitate reproducibility," their report asserts.

Journals and their publishers can also encourage improved reproducibility by insisting that workflows, data, and software to be part of the submission-and-review process, the authors say.

Finally, they note, better reproducibility may eventually be mandated, citing a recent administration memorandum asking all agencies to develop policies to make results of all federally funded research broadly available to scientists, industry, and the public.

Explore further: Scientific reproducibility is hampered by a lack of specificity of the material resources

Related Stories

New computational technique can predict drug side effects

December 11, 2007

Early identification of adverse effects of drugs before they are tested in humans is crucial in developing new therapeutics, as unexpected effects account for a third of all drug failures during the development process.

Recommended for you

New paper answers causation conundrum

November 17, 2017

In a new paper published in a special issue of the Philosophical Transactions of the Royal Society A, SFI Professor Jessica Flack offers a practical answer to one of the most significant, and most confused questions in evolutionary ...

Chance discovery of forgotten 1960s 'preprint' experiment

November 16, 2017

For years, scientists have complained that it can take months or even years for a scientific discovery to be published, because of the slowness of peer review. To cut through this problem, researchers in physics and mathematics ...

1 comment

Adjust slider to filter visible comments by rank

Display comments: newest first

1 / 5 (5) Nov 28, 2013
Very interesting that this same day, PhysOrg has another article on a test, evidently using unreliable anonymous information from anyone with a computer, that "reproduces" results in "psychology" that formerly were irreproducible. "Science" may be close to having it exposed that it's all hogwash. There, as here, "scientists" doing swindles in conventional areas have their frauds excused for "mistakes" rather than criminality on their part. When unconventional studies get reproducible results don't expect there necessarily won't be a "scientist" who says the reproducibility proves it's a lie! Why, for example, the issue of the scripts for the tuberculosis "experiment" not being available? Did they use a different definition for the mean or the standard deviation? All "science" is a scam.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.