Group calls for more transparency in science research, announces guidelines

June 25, 2015, Rice University
Credit: Charles Rondeau/public domain

An international group of academic leaders, journal editors and funding-agency representatives and disciplinary leaders, including Rick Wilson, the Herbert S. Autrey Chair of Political Science and professor of statistics and psychology at Rice University, has announced guidelines to further strengthen transparency and reproducibility practices in science research reporting.

The group, the Transparency and Openness Promotion (TOP) Committee at the Center for Open Science in Charlottesville, Va., outlined its new guidelines in a story published in this week's edition of the journal Science.

While transparency, openness and reproducibility are readily recognized as vital features of science and embraced by scientists as a norm and value in their work, a growing body of evidence suggests that those qualities are not necessarily evident today.

"A likely culprit for this disconnect is an academic reward system that insufficiently incentivizes open practices," Wilson said. "In the present reward system, the emphasis on innovation undermines practices that support openness. Too often, publication requirements—whether actual or perceived—fail to encourage transparent, open and reproducible science."

TOP's objective is to translate scientific norms and values into concrete actions and change the current incentive structures to drive researchers' behavior toward more openness.

"We know the disciplines differ in what is emphasized, so we sought to produce guidelines that focus on what is shared across disciplines," Wilson said.

Each of eight has three levels of adoption in the TOP guidelines; each moves scientific communication toward greater openness, according to the article in Science. These standards are modular, facilitating adoption in whole or in part. However, they also complement each other; commitment to one standard may facilitate adoption of others.

The standards include citation standards for journals, , analytic methods (code) transparency, research materials transparency, design and analysis transparency, preregistration of studies, preregistration of analysis plans and replication.

Two standards reward researchers for the time and effort they have spent engaging in open practices. Citation standards extend current article citation norms to data, code and research materials. Regular and rigorous citation of these materials credits them as original intellectual contributions. Replication standards recognize the value of replication for independent verification of research results and identify the conditions under which replication studies will be published in the journal.

Four of the standards describe what openness means across the scientific process so that research can be reproduced and evaluated. Reproducibility increases confidence in findings and also allows scholars to learn more about what results mean. Design standards increase transparency about the research process and reduce vague or incomplete reporting of the methodology. Standards for research materials encourage the provision of all elements of that methodology, and data-sharing standards give authors an incentive to make data available in trusted repositories.

The final two standards address the values resulting from preregistration. Standards for preregistration of studies facilitate the discovery of research, even unpublished research, by ensuring that the existence of the study is recorded in a public registry. Preregistrations of analysis plans certify the distinction between confirmatory and exploratory research, or what is also called hypothesis-testing versus hypothesis-generating research. Making the distinction between confirmatory and exploratory methods transparent can enhance .

"The guidelines are sensitive to concerns by both journals and researchers," Wilson said. "For example, we encourage journals to state exceptions to sharing because of ethical issues, intellectual property concerns or availability of necessary resources. We encourage journals to pick and choose among the different levels and standards in order to define what they expect of the researchers.

"We acknowledge the variation in evolving norms about research transparency. Depending on the discipline or publishing format, some of the standards may not be relevant for a journal. Journal and publisher decisions can be based on many factors—including their readiness to adopt modest to stronger transparency standards for authors, internal journal operations and disciplinary norms and expectations," Wilson said.

The present version of the guidelines is not the last word on standards for in science, according to the report. "As with any research enterprise, the available empirical evidence will expand with application and use of these guidelines," the TOP Committee wrote. "To reflect this evolutionary process, the guidelines are accompanied by a version number and will be improved as experience with them accumulates."

An information commons and support team at the Center for Open Science is available (top@cos.io) to assist journals in selection and adoption of standards and will track adoption across journals. Adopting journals may also suggest revisions that improve the guidelines or make them more flexible or adaptable for the needs of particular subdisciplines.

To read the complete guidelines, go to https://osf.io/ud578/.

Explore further: New gold standard established for open and reproducible research

More information: Promoting an open research culture, Science, www.sciencemag.org/lookup/doi/ … 1126/science.aab2374

Related Stories

No 'silver bullet' for science standards

April 18, 2013

America's K-12 teachers are not fully prepared to meet a new set of science standards, a Michigan State University education scholar argues in Science.

Call for better social science research transparency

January 2, 2014

In the Friday (Jan. 3) edition of the journal Science, an interdisciplinary group is calling on scholars, funders, journal editors and reviewers to adopt more stringent and transparent standards to give social science research ...

Recommended for you

Archaeologists discover Cornish barrow site

April 20, 2018

An Archaeologist at The Australian National University (ANU) has discovered a prehistoric Bronze-Age barrow, or burial mound, on a hill in Cornwall and is about to start excavating the untouched site which overlooks the English ...

New ancestor of modern sea turtles found in Alabama

April 18, 2018

A sea turtle discovered in Alabama is a new species from the Late Cretaceous epoch, according to a study published April 18, 2018 in the open-access journal PLOS ONE by Drew Gentry from the University of Alabama at Birmingham, ...

New study improves 'crowd wisdom' estimates

April 18, 2018

In 1907, a statistician named Francis Galton recorded the entries from a weight-judging competition as people guessed the weight of an ox. Galton analyzed hundreds of estimates and found that while individual guesses varied ...

10 comments

Adjust slider to filter visible comments by rank

Display comments: newest first

denglish
2.3 / 5 (3) Jun 25, 2015
I would like to see a challenge to the AGW world to adopt this standard.
COCO
not rated yet Jun 26, 2015
thin edge of the wedge here all - unless, as I suspect, the conclusions have been made beforehand
shavera
5 / 5 (5) Jun 26, 2015
denglish: it's not the scientists that are withholding their data. It's the fact that the "free-market" solution to communicating science has been for-profit journals. In order to fact-check and peer-review articles before publication, that costs money. So journals charge money for access. And since a journal article is usually only accessible to people with a high degree of scientific training to comprehend (one must use dense scientific jargon to compress an article into a reasonable reading length), the sales of such journals have naturally small audiences. Thus, the cost for each journal is fairly high.

I think what would be better, of course, is to publicly fund open-access journals and their editing. But I'm sure then you'd accuse *them* of operating with a pro-big-government bias.

So there are your options: expensive "free" journals, or free "public" journals.
antialias_physorg
5 / 5 (3) Jun 26, 2015
So there are your options: expensive "free" journals, or free "public" journals.


...or one could look for a preprint version on arxiv
...or one could write to the author and ask them for an electronic copy (this has worked for me 100% of the time. For some reason scientists actually *like* it if someone wants to read their paper. Go figure.)
antigoracle
1 / 5 (1) Jun 26, 2015
This would spell the end of the AGW Cult.
shavera
5 / 5 (2) Jun 26, 2015
Oh, yeah I agree that there are often ways "around" the journal system. But Arxiv doesn't work as a preprint server if there isn't, somewhere, some form of publication. You still need someone to do peer-review (maybe not the most perfect system, but better than nothing at all). Otherwise you get people publishing their "theories" online... *cough every crackpot ever cough*
shavera
5 / 5 (3) Jun 26, 2015
antigoracle: Why do you think that? Nearly all the scientists who currently have access to each others' data and methods agree on AGW. How would more people having access to the same information "end" that?

The anti-AGW side already cherry picks data and arguments. Would access to more data make them any less likely to cherry pick? Any less likely to take data out of context, or take analysis techniques out of scientific context? I doubt.

Open access won't do anything to change either side of the AGW debate. One side is scientific in nature already, and increased journal access won't help them. One side is economic and political in nature already, and increased science is irrelevant to their arguments in the first place.
gkam
1 / 5 (2) Jun 26, 2015
Did goricle answer the question of what caused the warming we had in the other thread? I think not.
antigoracle
1 / 5 (1) Jun 26, 2015
Shavera, perhaps you should have a look at the Climategate emails to see why I make that statement. Those emails are littered with admissions of how they would subvert the scientific process. The following are just a few:
- redefine what peer-review is
- destroy data, acquired with public funding, rather than make it available
- refuse to divulge their method
- threats to scientists and publishers who dare to have an opposing view

AGW is painted as such a global human tragedy, you would imagine that they would open their doors and beg for you to come in and have a look at their evidence.

For years I bought into the AGW scam, until I listened to a few scientists on the other side, and they all made one very compelling statement - Don't take my word for it, educate yourself and look at the evidence.
So, open access would let you see that evidence and not take either side just on their word.
antialias_physorg
5 / 5 (1) Jun 26, 2015
Oh, yeah I agree that there are often ways "around" the journal system. But Arxiv doesn't work as a preprint server if there isn't, somewhere, some form of publication. You still need someone to do peer-review (maybe not the most perfect system, but better than nothing at all).

I was just referring to papers that are published in peer reviewed journals. You can often get preprint versions of them at arxiv. Arxiv isn't a first go-to for research papers.
(Note that you should be *very* careful about citing an arxiv paper in your own publications. Citing the published version is infinitely preferrable).
Papers that appear only on arxiv and never anywhere else? Not a good source.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.