Opinion: Failure to reproduce results is 'a normal part of how science works'
Results of a new study in the prestigious journal Science hit the front page of The New York Times late last month. The study found that of 100 papers published in three top psychology journals, more than 60 could not be replicated—that is, the findings could not be reproduced. A storm of speculation ensued, with references to everything from fraud and exaggerated scientific claims to questions about whether the social sciences are even "science" in the first place.
Fortunately, calmer—and wiser—voices prevailed, including those of psychologists themselves, who welcomed the rigorous evaluation of the field. Lisa Feldman Barrett, University Distinguished Professor of Psychology at Northeastern, homed in on perhaps the most telling point the study revealed in an op-ed piece she penned for the New York Times: the importance of context. Who (human, animal, molecule) experiences what, and under which conditions, shapes an experiment's results.
For starters, the study, produced by a collaborative called the Reproducibility Project at the Center for Open Science, never said the results of the original experiments were wrong or even inflated; it said they did not hold up on a second try.
"But the failure to replicate is not a cause for alarm," noted Barrett sagely. "In fact, it is a normal part of how science works."
Researchers from fields as diverse as molecular biology and electrical engineering know that. Indeed, Lloyd Sederer, medical director of the New York State Office of Mental Health, mused in a U.S. News and World Report blog whether the Reproducibility Project's results "would themselves stand the test of replication."
Science, wrote Barrett, relies on trial, results, validation of results, and if not, clues to next steps. She cited famous studies in genetics, physics, and psychology that failed to replicate to underline how a particular phenomenon may be true but "only under certain conditions." If those conditions are altered, even slightly, the results will likely change. Remember the dress that sprang into meme-land across social media in February? Was it black and blue or white and gold? Well, it depended on conditions: in this case, lighting conditions—how each viewer's brain interpreted the image's ambiguous lighting.
"Science is not a body of facts that emerge, like an orderly string of light bulbs, to illuminate a linear path to universal truth," Barrett concluded. "Rather, science (to paraphrase Henry Gee, an editor at Nature) is a method to quantify doubt about a hypothesis, and to find the contexts in which a phenomenon is likely. Failure to replicate is not a bug; it is a feature. It is what leads us along the path—the wonderfully twisty path—of scientific discovery."