The replication crisis: Researchers reveal a hidden universe of uncertainty

Researchers reveal a hidden universe of uncertainty
Broad variation in the findings from 73 teams testing the same hypothesis with the same data. The distribution of estimated AMEs across all converged models (n = 1,253) includes results that are negative (yellow; in the direction predicted by the given hypothesis the teams were testing), not different from zero (gray), or positive (blue) using a 95% CI. AME are xy standardized. The y axis contains two scaling breaks at ±0.05. Numbers inside circles represent the percentages of the distribution of each outcome inversely weighted by the number of models per team. Credit: Proceedings of the National Academy of Sciences (2022). DOI: 10.1073/pnas.2203150119

The University of Luxembourg's Department of Social Sciences contributed to a large-scale replication study that aimed to understand the role of decisions that scientists make during the research process. Published in the Proceedings of the National Academy of Sciences, the study highlights the importance of open science and collaboration among scientists.

During the COVID 19 pandemic, a certain crisis of credibility in science manifested itself. It has also become apparent that both the public and experts have difficulty dealing with scientific uncertainty.

Part of this mix is the "": In several , studies have repeatedly failed to replicate a significant proportion of previously published results. Even if this can be explained by errors, publication pressure and bias or even questionable research practices, the failure of replications undermines the role of science as a reliable producer of knowledge.

A study led by Nate Breznau, Eike Mark Rinke, Alexander Wuttke and colleagues investigated the role of decisions that scientists make during the . Andreas Hadjar and Robin Samuel from the Department of Social Sciences at the Faculty of Humanities, Education and Social Sciences of the University of Luxembourg contributed to this large-scale replication study, which involved 161 researchers in 73 teams from around the world.

One hypothesis, one data set, various results

The focus of the replication was to test the hypothesis that more immigration reduces for government provision of social measures.

Interestingly, the 1,253 statistical models contributed by the research teams produced very different results—even though they used the same data. The analytical choices, the expertise of the researchers and their expectations could not explain these huge differences in the results.

The authors concluded that a "hidden universe of uncertainty remains." Consequently, "scientists, especially those working with the complexities of human societies and behavior, should exercise humility and strive to better account for the uncertainty in their work." The authors further highlight the potential and importance of transparent and .

More information: Breznau, Rinke, Wuttke et al, Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty, Proceedings of the National Academy of Sciences (2022). DOI: 10.1073/pnas.2203150119

Citation: The replication crisis: Researchers reveal a hidden universe of uncertainty (2022, November 28) retrieved 1 February 2023 from https://phys.org/news/2022-11-replication-crisis-reveal-hidden-universe.html
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.

Explore further

A new replication crisis: Research that is less likely to be true is cited more

9 shares

Feedback to editors