ASCB task force on scientific reproducibility calls for action and reform

In the face of growing concerns about the reproducibility of published scientific data, a special task force of the American Society for Cell Biology has made 13 recommendations to tighten standards, improve statistics and ethics training, and encourage self-policing by life scientists.

"We ignore these reproducibility issues at our own peril," warns a new white paper issued today by a special task force of the American Society for Cell Biology. After a year's study of the problems surrounding reproducibility of experimental results in life science research, the ASCB Data Reproduction Task Force made 13 recommendations to tighten standards in labs, federal agencies, research institutions, and scientific journals. The task force strongly endorsed self-policing by communities of working scientists to establish reproducibility benchmarks for methods, data, and assays within their fields. The ASCB itself, says the task force, must work closely with the National Institutes of Health (NIH) and other scientific standards groups to authenticate cell lines and other materials used in labs and to emphasize experimental reproducibility as a core principle in training young scientists.

In addition, the white paper described a survey of ASCB's 8,000 members in which nearly three quarters of the 869 respondents said they'd had trouble replicating another lab's published results while one third said that they themselves had received reports of difficulties in replicating one of their own published experiments from another lab. Sixty percent said they'd resolved such questions through "amicable" communication and only 18% said that problems were left unresolved. While the ASCB task force was forced to call the survey "qualitative, not quantitative" because the response rate of 10.8% was lower than could be considered statistically valid, the members felt the results indicated how seriously ASCB scientists take reproducibility problems. Even in these days of tight funding, 54% of respondents said their labs would invest significant time and money to resolve a reproducibility question.

Public concern about scientific practice is appropriate, the ASCB task force declared. "The citizens of the United States have provided many years of significant support for basic life sciences research and they have the right to expect scientific results that can be reproduced and built upon to create a better understanding of biological processes and disease states." The task force cited news media reports, scientific analyses, and policy commentaries that describe how often other labs in the field could not replicate published scientific results. The heart of the reproducibility problem is not outright fraud or misconduct, says the ASCB task force, but a combination of misinformation, particularly in training scientists in statistical analysis, incomplete information in published papers about methods and materials, as well as a raft of academic and economic pressures.

One of the task force's strongest recommendations was to promote scientific community-based standards for assessing data from a given field. Research biology today is self-divided into small fields of study where researchers know each other as collaborators and competitors. The report singles out the efforts of Daniel Klionsky of the University of Michigan who is editor of the journal, Autophagy, a field that studies how cells self-degrade their contents and recycle the materials for new cells. Concerned about differing standards of proof among autophagy labs, Klionsky spearheaded a community effort to draw up specific standards for proof, which are now widely accepted by researchers and journal editors in the field. The ASCB task force thought that other scientific societies and journals should encourage researchers in the fields they serve to cooperate in similar standard-setting efforts.

Other recommendations of the report include support for NIH-led efforts to improve training in statistics and scientific ethics such as the National Institute for General Medical Sciences' Training Modules to Enhance Data Reproducibility. It also endorsed work by agencies such as NIH and non-profits such as the Global Biological Standards Institute to authenticate cell lines for laboratory use.

The task force was especially critical of the scientific publishing industry for creating a climate in which researchers desperate to place their work in a few "high impact" journals believe they must cherry pick data or exaggerate the implications of limited experiments. The journals themselves were faulted for truncated and rushed peer review as well for arbitrary limits on methods documentation and supporting materials. The task force reiterated its opposition to journal-based impact metrics as a major distortion in science publishing, urging wider application of the San Francisco Declaration on Research Assessment (DORA) recommendations to curtail their use. In the new age of Big Data biology, the Task Force urged all parties to make raw data and analytic algorithms fully accessible.

Led by Mark Winey, PhD, of the University of Colorado, Boulder, the task force included ASCB Executive Director Stefano Bertuzzi, PhD, Nobel laureate Carol Greider, PhD, Johns Hopkins University, Doug Koshland, Ph.D. University of California, Berkeley, Connie Lee, Ph.D., University of Chicago, Paul Mungai, PhD, American Association Advancement of S cience Policy Fellow, and Brian Nosek, PhD, Center for Open Science.