Inflation of online ratings can be beneficial and detrimental

Many adults consult online ratings and reviews before they make a purchase. However, how ratings affect sales is a challenging topic as evidenced from controversy surrounding Rotten Tomato scores or sellers trying to manipulate ratings on various platforms. A new study investigates how rating inflation affected a digital platform as well as the choices users made. The study found that inflation can have benefits and detriments: While sales rose, users' trials declined, and sales were concentrated among popular sellers. Rating inflating is when platforms change their strategy to increase average ratings. So, users see higher ratings for all restaurants.

The study, by researchers at Carnegie Mellon University (CMU) and the University of British Columbia, appears in Information Systems Research.

"Overall, our results illustrate the potential consequences of rating inflation that platforms need to consider when designing and managing their rating systems," says Rahul Telang, professor of information systems and management at CMU's Heinz College, who coauthored the study.

In the process of rating inflation, the variance across restaurants do down—since restaurants tend to have higher ratings, they look more similar. For instance, on eBay, the median seller might have a 100% positive rating, while the seller in the bottom 10th percentile has a 98% positive rating. Not only are these exceptionally high positive ratings, but there is little variation between good and mediocre sellers, making the overall ratings less informative for consumers.

In this study, researchers conducted an experiment with a digital food-delivery platform in a large Asian city that changed its rating in a neighborhood (with 48 restaurants on the platform), which resulted in rating inflation. They examined the impact of this change over 13 weeks in mid-2017 on users' purchases, trials of new restaurants, and sales concentration in the context of choosing restaurants on the platform.

The study found that the platform benefitted from one aspect of rating inflation: Because this type of inflation makes sellers' products appear to be of higher quality, both users' purchases and sellers' sales increased in the short term due to increased average ratings.

However, there were also negative consequences: The decrease in rating variance reduced the informativeness of ratings, which increased the uncertainty of users' perceptions of restaurants' quality and made them less likely to try a new establishment. Since perceptions of ratings were less certain, they became less important in shaping users' perceptions of quality, and this boosted the importance of other signals (e.g., prior beliefs, prior experience), leading to a greater concentration of sales among popular restaurants.

Among the study's limitations, the authors note they did not investigate the long-term effects of rating inflation. In addition, their data are from one city, limiting the generalizability of their results.

"Our findings offer important insights about the tradeoffs of rating inflation for managers, designers, and developers of digital platforms that use ratings to help users choose among numerous sellers," suggests Hui Li, associate professor of marketing at CMU's Tepper School of Business, who coauthored the study. "Such individuals should build informativeness into the design of rating systems to account for and minimize rating inflation on their platforms."