The irreproducibility crisis – an opportunity to make science better

Is there a reproducibility crisis in science? A recent news feature published in Nature is only the latest piece to suggest that there may be.

Among the 1,576 researchers surveyed in this news feature, 52% noted that reproducibility is a significant crisis in science. Physicists and chemists had the greatest confidence in their respective fields while medical professionals and biologists had the least. In addition, the survey found that 24% and 13% of respondents had published successful and unsuccessful replications, respectively, compared to only 12% and 10% of those whose findings were rejected. These findings are similar to previous studies that found that only 16 of 83 articles recommending the effectiveness of various psychiatric treatments were successfully replicated and that only 36% of replication studies among 100 experimental and correlational studies published in three psychology journals were successfully reproduced.

Why is reproducibility vital in scientific research?

Simply put, reproducibility is the ability to generate similar results each time an experiment is duplicated, either by the same researcher or by a different one. Given that science aims to generate findings that could enhance our understanding of the world, reproducibility not only ensures the value of the findings in a manuscript, but also prevents other researchers from being led down blind alleys.

While reproducibility is at the heart of science, failure to do so is prevalent in research. Being at the cutting edge of science sometimes fuels the pressure of publishing novel results so that scientists can advance further in their fields. Yet, this pressure should not be the main engine driving scientific research as it could potentially generate too many false positive results. In fact, according to a June 2015 PLOS Biology article, based on an analysis of previous studies, the prevalence of irreproducible preclinical research exceeds 50% and has a hefty price tag of approximately $28.2 billion. In addition, this issue could contribute to the public's mistrust of scientists and the scientific process, which would threaten the future of scientific progress.

Causes of irreproducibility in science

In addition to pressures of publishing in science, there are many other possible factors that can cause irreproducibility. Selective reporting of results, a lack of understanding of fundamental statistical principles, and poor experimental design all contribute to this pervasive dilemma. Other factors include insufficient mentoring from senior scientists, fraud, hyper-competition between lab members in a publish-or-perish environment, and insufficient resources to conduct the research.

Another principal factor that could cause this dilemma is the variability of experiments performed among different labs that answer similar research questions. One common example is the use of cell lines to understand fundamental biological processes and disease states. In a June 2016 PLOS Biology paper, Capes-Davis and Neve suggested that, since reproducibility in the biomedical sciences has been called into question, quality assurance of cell lines should be implemented to authenticate these resources. They suggested that training on authentication of molecular biology resources to early career researchers and placing a greater emphasis on focused, good science rather than publications or big science could lessen the crisis. In addition, in a recently published PLOS Biology paper, Almeida et al. suggested that community action regarding how to utilize cell lines properly and the existence of various authentication standards, such as the ANSI/ATCC standard (ASN-0002) for STR profiling and ASN-0003 for species barcoding, are key steps for ensuring the quality of cell lines.

The lack of detailed experimental methods in published studies could also contribute to irreproducibility. If researchers put more effort into meticulously describing their complicated experiments, perhaps other scientists would find it easier to replicate them. Many journals are now implementing specific rules to ensure that authors provide all the information that is needed to reproduce their results. Furthermore, some journals, such as Nature, have generated open repositories where researchers can deposit and share their protocols with the rest of the scientific community for use and comment.

What else could be done?

It is not entirely clear how science can be made more reproducible. However, respondents from the Nature editorial suggested that better understanding of statistical principles, better mentoring, more robust experimental design, more within-lab validation, and better teaching are potential methods for improving reproducibility. Moreover, journals, funding organizations, and research institutions should adopt stricter standards for paper and protocol acceptance, even at the expense of increased peer review time, so that more researchers would adopt this practice and provide more confidence to their scientific discoveries.

In addition to these recommendations, I think that more effective collaboration among scientists performing similar research topics could address this issue. However, these initial collaborative conversations may be difficult. At present, less than 20% of researchers said they had been contacted by another researcher who was unable to reproduce their work. Researchers who reach out to other researchers risk appearing incompetent, accusatory, or disclosing too much about their own projects. While this method may appear nearly impossible, it may provide a forum for which researchers could rigorously validate their work and advance further in their research careers.

While complete reproducibility may not be possible for scientific research, I encourage early career researchers to ponder how they could improve the reproducibility of their own research. Although replicating results may require more time and resources, given that most experimental results reported in the literature will not be subjected to rigorous replication unless they are challenged, it is essential for investigators to put their best efforts in making their results as robust as possible. Reproducibility remains central to science, and the consequences for irreproducible research are detrimental and deserve immediate attention.

This story is republished courtesy of PLOS Blogs: blogs.plos.org.