Aiming for gold—improving reproducibility in hydrology studies

Aiming for gold: improving reproducibility in hydrology studies
Researchers at Utah State University are creating online tools to help make hydrology studies more reproducible. Credit: Matt Jensen

In six well-regarded hydrology and water resources journals published in 2017, the estimated percentage of studies whose results could be fully reproduced was only between 0.06 and 6.8 percent. This low level of reproducibility is not uncommon in hydrology studies—a fact many scientists readily acknowledge. However, a team of researchers at Utah State University may have found a solution to make these studies more reproducible.

In their paper, "Assessing data availability and research reproducibility in hydrology and water resources," published Feb. 26 in Nature's Scientific Data, David Rosenberg co-authors developed an online to assess the reproducibility of published research. The team reviewed 360 articles from six water resource journals published in 2017. Out of 360 articles, they could only fully reproduce results from four articles.

"Our survey tool breaks down the concept of scientific reproducibility into specific components of data availability, reproducibility of results and replicability of findings," said Rosenberg, an associate professor of civil and environmental engineering at USU. "We then suggest how authors, journals, funders and institutions can use the survey tool to increase low rates of reproducibility."

The authors say reproducibility can be broken down into three components:

  • Are the data, models, code, directions for use and other artifacts used in the work available?
  • Can artifacts be used to reproduce published results?
  • Can the findings be replicated with new datasets?

The team's online survey tool consists of 15 questions and provides a checklist of the essential items needed for availability and result reproducibility. Artifacts is an umbrella term for all data, software, models, code, directions and other materials needed to reproduce results within a study.

The team found that about 70 percent of the sampled articles stated some materials were available but only around 48 percent of the materials could be accessed online. Only around six percent of the sampled articles made artifacts publicly available, and only one percent of sampled articles made artifacts available and could be fully reproduced.

The authors said many articles were missing directions to generate results. If authors provide directions, they say, the number of articles that could be tested for reproducibility would double. Articles that made all artifacts available had a six in 10 chance of having some or all of their results reproduced. Two journals surveyed by the team required articles to state how artifacts can be accessed, and four journals encouraged statements. No journals required authors to make all artifacts available.

The survey tool can help recognize and encourage authors to achieve certain reproducibility levels. For example, authors can use the survey tool to self-assess the reproducibility of their results. Rosenberg and his team also recommend a medal system to recognize different levels of reproducibility:

  • Bronze Medal: All artifacts are made available within the or in open repositories
  • Silver Medal: All artifacts are made available and results are fully reproducible
  • Gold Medal: Results are fully reproducible and overall findings can be replicated in different settings with the same or different artifacts

Rosenberg recommend that medal icons be posted next to online articles to recognize authors for their reproducibility work and make it easier for readers to find top reproducibility practices. Rosenberg and his team assigned four silver medals and six bronze medals out of the 360 articles they reviewed. Awarding gold medals for replicability of findings remains an important line of future work.

"We hope the survey tool will help nudge authors, journals, funders and institutions to make scientific work more reproducible," said Rosenberg. "We welcome discussion to improve the survey tool and to improve the of our science."


Explore further

Transparency and reproducibility of biomedical research is improving

More information: James H. Stagge et al, Assessing data availability and research reproducibility in hydrology and water resources, Scientific Data (2019). DOI: 10.1038/sdata.2019.30
Citation: Aiming for gold—improving reproducibility in hydrology studies (2019, February 27) retrieved 21 May 2019 from https://phys.org/news/2019-02-aiming-goldimproving-hydrology.html
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.
7 shares

Feedback to editors

User comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more