Opinion: Failure to reproduce results is 'a normal part of how science works'

September 11, 2015 by Thea Singer, Northeastern University

Results of a new study in the prestigious journal Science hit the front page of The New York Times late last month. The study found that of 100 papers published in three top psychology journals, more than 60 could not be replicated—that is, the findings could not be reproduced. A storm of speculation ensued, with references to everything from fraud and exaggerated scientific claims to questions about whether the social sciences are even "science" in the first place.

Fortunately, calmer—and wiser—voices prevailed, including those of psychologists themselves, who welcomed the rigorous evaluation of the field. Lisa Feldman Barrett, University Distinguished Professor of Psychology at Northeastern, homed in on perhaps the most telling point the study revealed in an op-ed piece she penned for the New York Times: the importance of context. Who (human, animal, molecule) experiences what, and under which conditions, shapes an experiment's results.

For starters, the study, produced by a collaborative called the Reproducibility Project at the Center for Open Science, never said the results of the original experiments were wrong or even inflated; it said they did not hold up on a second try.

"But the failure to replicate is not a cause for alarm," noted Barrett sagely. "In fact, it is a normal part of how science works."

Researchers from fields as diverse as molecular biology and electrical engineering know that. Indeed, Lloyd Sederer, medical director of the New York State Office of Mental Health, mused in a U.S. News and World Report blog whether the Reproducibility Project's results "would themselves stand the test of replication."

Science, wrote Barrett, relies on trial, results, validation of results, and if not, clues to next steps. She cited famous studies in genetics, physics, and psychology that failed to replicate to underline how a particular phenomenon may be true but "only under certain conditions." If those conditions are altered, even slightly, the results will likely change. Remember the dress that sprang into meme-land across social media in February? Was it black and blue or white and gold? Well, it depended on conditions: in this case, lighting conditions—how each viewer's brain interpreted the image's ambiguous lighting.

"Science is not a body of facts that emerge, like an orderly string of light bulbs, to illuminate a linear path to universal truth," Barrett concluded. "Rather, science (to paraphrase Henry Gee, an editor at Nature) is a method to quantify doubt about a hypothesis, and to find the contexts in which a phenomenon is likely. Failure to replicate is not a bug; it is a feature. It is what leads us along the path—the wonderfully twisty path—of scientific discovery."

Explore further: Massive study reports challenges in reproducing published psychology findings

More information: "Estimating the reproducibility of psychological science." Science 28 August 2015: DOI: 10.1126/science.aac4716

Related Stories

Video: The basics of reproducibility

September 2, 2015

The scientific community is constantly working to improve the robustness and reliability of published research. Brain Nosek, president and director of the Center for Open Science, has dedicated his career to asking—and ...

Psychologists collaborate to reproduce experimental results

November 28, 2013

(Medical Xpress)—Psychological studies have a bad reputation for delivering results that researchers never reproduce. Repeatedly, psychologists have been unable to replicate the effects of classic studies. To test the reproducibility ...

Recommended for you

Study sheds new light on ancient human-turkey relationship

January 17, 2018

For the first time, research has uncovered the origins of the earliest domestic turkeys in ancient Mexico. The study also suggests turkeys weren't only prized for their meat—with demand for the birds soaring with the Mayans ...

Lifting barriers to citizenship for low-income immigrants

January 15, 2018

Taking the Oath of Allegiance at a naturalization ceremony is an emotional moment for many immigrants, and for good reason: it is the culmination of an often arduous process and many years of striving. Citizenship also opens ...

1 comment

Adjust slider to filter visible comments by rank

Display comments: newest first

Squirrel
not rated yet Sep 12, 2015
The original papers--if they are typical of those in psychology journals--did not make such qualifications. They made claims that are now shown to be considerably weaker than when they put out. There is a problem.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.