Opinion: Failure to reproduce results is 'a normal part of how science works'

September 11, 2015 by Thea Singer, Northeastern University

Results of a new study in the prestigious journal Science hit the front page of The New York Times late last month. The study found that of 100 papers published in three top psychology journals, more than 60 could not be replicated—that is, the findings could not be reproduced. A storm of speculation ensued, with references to everything from fraud and exaggerated scientific claims to questions about whether the social sciences are even "science" in the first place.

Fortunately, calmer—and wiser—voices prevailed, including those of psychologists themselves, who welcomed the rigorous evaluation of the field. Lisa Feldman Barrett, University Distinguished Professor of Psychology at Northeastern, homed in on perhaps the most telling point the study revealed in an op-ed piece she penned for the New York Times: the importance of context. Who (human, animal, molecule) experiences what, and under which conditions, shapes an experiment's results.

For starters, the study, produced by a collaborative called the Reproducibility Project at the Center for Open Science, never said the results of the original experiments were wrong or even inflated; it said they did not hold up on a second try.

"But the failure to replicate is not a cause for alarm," noted Barrett sagely. "In fact, it is a normal part of how science works."

Researchers from fields as diverse as molecular biology and electrical engineering know that. Indeed, Lloyd Sederer, medical director of the New York State Office of Mental Health, mused in a U.S. News and World Report blog whether the Reproducibility Project's results "would themselves stand the test of replication."

Science, wrote Barrett, relies on trial, results, validation of results, and if not, clues to next steps. She cited famous studies in genetics, physics, and psychology that failed to replicate to underline how a particular phenomenon may be true but "only under certain conditions." If those conditions are altered, even slightly, the results will likely change. Remember the dress that sprang into meme-land across social media in February? Was it black and blue or white and gold? Well, it depended on conditions: in this case, lighting conditions—how each viewer's brain interpreted the image's ambiguous lighting.

"Science is not a body of facts that emerge, like an orderly string of light bulbs, to illuminate a linear path to universal truth," Barrett concluded. "Rather, science (to paraphrase Henry Gee, an editor at Nature) is a method to quantify doubt about a hypothesis, and to find the contexts in which a phenomenon is likely. Failure to replicate is not a bug; it is a feature. It is what leads us along the path—the wonderfully twisty path—of scientific discovery."

Explore further: Massive study reports challenges in reproducing published psychology findings

More information: "Estimating the reproducibility of psychological science." Science 28 August 2015: DOI: 10.1126/science.aac4716

Related Stories

Video: The basics of reproducibility

September 2, 2015

The scientific community is constantly working to improve the robustness and reliability of published research. Brain Nosek, president and director of the Center for Open Science, has dedicated his career to asking—and ...

Psychologists collaborate to reproduce experimental results

November 28, 2013

(Medical Xpress)—Psychological studies have a bad reputation for delivering results that researchers never reproduce. Repeatedly, psychologists have been unable to replicate the effects of classic studies. To test the reproducibility ...

Recommended for you

University choice and achievement partly down to DNA

October 18, 2018

Research from King's College London has shown for the first time that genetics plays a significant role in whether young adults choose to go to university, which university they choose to attend and how well they do.

1 comment

Adjust slider to filter visible comments by rank

Display comments: newest first

not rated yet Sep 12, 2015
The original papers--if they are typical of those in psychology journals--did not make such qualifications. They made claims that are now shown to be considerably weaker than when they put out. There is a problem.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.