Experimental economics: Results you can trust

March 3, 2016

Reproducibility is an important measure of validity in all fields of experimental science. If researcher A publishes a particular scientific result from his laboratory, researcher B should be able to follow the same protocol and achieve the same result in her laboratory. However, in recent years many results in a variety of disciplines have been questioned for their lack of reproducibility. A new study suggests that published results from experimental economics—a field pioneered at Caltech—are better than average when it comes to reproducibility.

The work was published in the March 3 online issue of the journal Science.

"Trying to reproduce previous results is not glamorous or creative, so it is rarely done. But being able to get the same result over and over is part of the definition of what makes knowledge scientific," says Colin Camerer, the Robert Kirby Professor of Behavioral Economics at Caltech and lead author on the paper.

The study was based on a previous method used to assess the replication of psychology experiments. In the earlier technique, called the reproducibility project psychology (RPP), researchers replicated 100 original studies published in three of the top journals in psychology—and found that although 97 percent of the original studies reported so-called "positive findings" (meaning a significant change compared to control conditions), such positive findings were reliably reproduced only 36 percent of the time.

Using this same technique, Camerer and his colleagues reproduced 18 laboratory experimental papers published in two top-tier economics journals between 2011 and 2014. Eleven of the 18—roughly 61 percent—showed a "significant effect in the same direction as in the original study." The researchers also found that the sample size and p-values—a standard measure of statistical confidence—of the original studies were good predictors for the success of replication, meaning they could serve as good indicators for the reliability of results in future experiments.

"Replicability has become a major issue in many sciences over the past few years, with often low replication rates," says paper coauthor Juergen Huber of the University of Innsbruck. "The rate we report for experimental economics is the highest we are aware of for any field."

The authors suggest that there are some methodological research practices in laboratory experimental economics that contribute to the good replication success. "It seems that the culture established in experimental economics—incentivizing subjects, publication of the experimental procedure and instructions, no deception—ensures reliable results. This is very encouraging given that it is a very young discipline," says Michael Kirchler, another coauthor and collaborator from the University of Innsbruck.

"As a journal editor myself, we are always curious whether experimental results will replicate across populations and cultures, and these results from multiple countries are really reassuring," says coauthor Teck-Hua Ho from the National University of Singapore.

Coauthor Magnus Johannesson from the Stockholm School of Economics adds, "It is extremely important to investigate to what extent we can trust published scientific findings and to implement institutions that promote scientific reproducibility."

"For the past half century, Caltech has been a leader in the development of social science experimental methods. It is no surprise that Caltech scholars are part of a group that use replication studies to demonstrate the validity of these methods," says Jean-Laurent Rosenthal, the Rea A. and Lela G. Axline Professor of Business Economics and chair of the Division of the Humanities and Social Sciences at Caltech.

Explore further: Psychologists collaborate to reproduce experimental results

More information: "Evaluating replicability of laboratory experiments in economics," Science (2016). DOI: 10.1126/science.aaf0918

Related Stories

Psychologists collaborate to reproduce experimental results

November 28, 2013

(Medical Xpress)—Psychological studies have a bad reputation for delivering results that researchers never reproduce. Repeatedly, psychologists have been unable to replicate the effects of classic studies. To test the reproducibility ...

Video: The basics of reproducibility

September 2, 2015

The scientific community is constantly working to improve the robustness and reliability of published research. Brain Nosek, president and director of the Center for Open Science, has dedicated his career to asking—and ...

Recommended for you

Ancient Egyptians used metal in wooden ships

August 31, 2016

A piece of wood recovered at a dig near the Great Pyramid of Giza shows for the first time that ancient Egyptians used metal in their boats, archaeologists said Wednesday.

Reconstructing the sixth century plague from a victim

August 30, 2016

Before the infamous Black Death, the first great plague epidemic was the Justinian plague, which, over the course of two centuries, wiped out up to an estimated 50 million (15 percent) of the world's population throughout ...

New species of pterosaur discovered in Patagonia

August 30, 2016

Scientists today announced the discovery of a new species of pterosaur from the Patagonia region of South America. The cranial remains were in an excellent state of preservation and belonged to a new species of pterosaur ...

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.