Airbnb study sheds light on whether incentivizing customer reviews makes a difference

New research focused on whether incentivizing would-be online reviewers to post reviews can make a difference for the seller has shed new light on the role those reviews play in sales and revenue outcomes.

Researchers worked with Airbnb to conduct a large-scale and costly experiment to find out just how those incentivized reviews impact day-to-day marketing. They found that although incentivizing reviews does lead to an increased number of reviews, the induced reviews tend to be more negative in nature, and have no impact on the number of nights sold or total revenue.

The study published in the journal Marketing Science, "Do Incentives to Review Help the Market? Evidence from a Field Experiment on Airbnb," is authored by Andrey Fradkin of Boston University and David Holtz of the University of California, Berkeley, and the MIT Initiative on the Digital Economy.

The Airbnb experiment divided participants into two groups: treatment and control. The treatment group was sent an email offering them $25 Airbnb coupons in exchange for a review if they had not already posted a review within the typical eight or nine days after checkout. The control group was sent an email simply reminding them to leave a review with no financial incentive offered.

"We found that reviews induced by incentives were more likely to contain information about lower quality experiences," says Fradkin. "We also found that when compared to non-incentivized reviews with the same star ratings, incentivized reviews corresponded to worse guest experiences. In other words, a seller may have received the same star rating from both study groups, but the incentivized guest was more likely to post a negative review, while upholding an inflated star rating."

"While the incentivized review program impacted the number and type of reviews left by guests, it had neutral-to-negative impacts on the actual business outcomes of the platform and its sellers," says Holtz.

"Incentivized reviews did not affect the total quantity (nights) sold for study listings, but they did cause a change in the composition of those transactions. More to the point, incentivized listings led to more transactions, but those transactions lasted fewer nights on average. Thus, a poorer match."

The study found that as a consequence, the revenue impacts of incentivized reviews did not represent either a positive or negative material change. Incentivized reviews failed to improve transaction quality, and according to some measures, resulted in worse matches. At the same time, the incentivized treatment did not lead to higher complaint rates or reduced likelihood of repeat business.

"Ultimately, we found that while reviews do play a meaningful role in the overall marketing efforts for online sellers, incentivizing those reviews may not deliver the results many would assume," says Fradkin.