Algorithmic recommendation technology or human curation? Study of online news outlet suggests both

Recommender systems are machine learning applications in online platforms that automate tasks historically done by people. In the news industry, recommender algorithms can assume the tasks of editors who select which news stories people see online, with the goal of increasing the number of clicks by users, but few studies have examined how the two compare.

A new study examined how users of an online news outlet in Germany reacted to automated recommendations versus choices made by human editors. On average, the algorithm outperformed the person, but the person did better under certain conditions. The study's authors suggest a combination of human curation and automated recommender technology may be best.

The study was conducted by researchers at Carnegie Mellon University (CMU), the University of Lausanne, and Ludwig-Maximilians-Universität (LMU) München. It is published in Management Science.

"Our work highlights a critical tension between detailed yet potentially narrow information available to algorithms and broad but often unscalable information available to humans," explains Ananya Sen, assistant professor of information systems and economics at CMU's Heinz College, who coauthored the study. "Algorithmic recommendations personalize at scale using information that tends to be detailed but is often temporally narrow and context-specific, while human experts base recommendations on broad knowledge accumulated over a professional career but cannot make individual recommendations at scale."

To quantify how companies should use algorithmic recommendation technology relative to human curation, researchers studied users' reactions to automated recommendations compared to how they reacted to human recommendations at a major online news outlet in Germany from December 2017 to May 2018. The outlet is an ad-supported publisher with more than 20 million monthly visitors and nearly 120 million monthly page impressions.

On average, the algorithmic recommendations outperformed those curated by human editors with respect to users' clicks. But this result depended on the experience of the human editors (more experienced editors did better than less experienced editors), the amount of personal data available to the algorithm (the algorithm required sufficient volume to perform well), and variation in the external environment that caused variation in demand for articles (humans did better on days with more attention-grabbing news).

The findings suggest that reverting to human curation can mitigate the drawbacks of personalized algorithmic recommendations, the authors say. They also suggest that platforms should defer to human expertise in the absence of user-specific personal data. The optimal combination of human curation and automated recommendation technology can lead to an increase of up to 13% in clicks.

"Based on our experiment, we suggest that managers leverage humans and automatic recommendations together rather than looking at curation as an issue that pits human experts against algorithms," says Christian Peukert, professor of strategy, globalization, and society at the University of Lausanne's business school, who co-authored the study.

Among the study's limitations, the authors say their experiment tested only how one algorithm performed relative to human editors, so their findings may apply only to news media that is supported by ads.