In measuring how online news influence political views, do mouse clicks speak louder than words?

In a polarized country, how much does the media influence people's political views? A new study co-authored by MIT scholars finds the answer depends on people's media preferences—and, crucially, how these preferences are measured.

The researchers combined a large online survey experiment with web-tracking data that recorded all of the news sites participants visited in the month before the study. They found that the media preferences individuals reported in the survey generally mirrored their real-world news consumption, but important differences stood out.

First, there was substantial variation in the actual news consumption habits of participants who reported identical media preferences, suggesting that survey-based measures may not fully capture the variance in individuals' experiences. Additionally, people with divergent media preferences in the survey often visited similar online news outlets. These findings challenge common assumptions about the polarized nature of Americans' media habits and raise questions about the use of survey data when studying the effects of political media.

"There's good reason to think that the information people report in surveys may not be a perfect representation of their actual media habits," says Chloe Wittenberg Ph.D. '23, a postdoc in the MIT Department of Political Science and co-author of a new paper detailing the results.

The open-access paper, "Media Measurement Matters: Estimating the Persuasive Effects of Partisan Media with Survey and Behavioral Data," appears in the Journal of Politics. The authors are Wittenberg; Matthew A. Baum, a professor at the Harvard Kennedy School; Adam Berinsky, MIT's Mitsui Professor of Political Science and director of the MIT Political Experiments Research Lab; Justin de Benedictis-Kessner, an assistant professor of public policy at the Harvard Kennedy School; and Teppei Yamamoto, a professor of political science and director of MIT's Political Methodology Lab.

Stated and revealed preferences

The study was motivated by a split within some academic research. Some scholars believe existing polarization produces highly partisan media consumption; others think partisan media sources influence citizens to adopt more polarized views. But few have measured both self-selection of media and its persuasive effects at the same time—using both survey and behavioral data.

To conduct the experiment, the researchers contracted with the media analytics company comScore to recruit a diverse sample of American adults in 2018. ComScore then combined survey responses from over 3,300 of these participants with detailed information about their web-browsing history in the month prior to the study.

"In this study, we adopted a novel experimental design called the Preference-Incorporating Choice and Assignment design—or the PICA design—which we invented and derived a formal statistical framework for in an earlier work," Yamamoto says. "The PICA design was a perfect fit for the study, given its objectives."

In the first part of the experiment, participants were asked to report their media preferences, including the quantity and type of news they like to read. In the second part, participants were assigned to one of two groups. The first group could select which type of media—Fox News, MSNBC, or an entertainment option—they wanted to read, whereas the second group was required to view articles from one of these three sources.

This approach enabled the researchers to assess both how individuals' stated preferences in the survey compared to their online news consumption, and how persuasive partisan media can be to different sets of consumers.

Overall, the study revealed differences in the persuasiveness of partisan media across news audiences. When examining the volume of news that participants consumed, the authors found that people who generally visited fewer news sites, relative to entertainment sites, tended to be more readily persuaded by partisan media.

However, when they looked at the political slant of participants' news consumption, the authors observed a small but striking deviation between their survey and behavioral measures of media preferences. At one end, the results based on survey data suggested that members of the public may be receptive to information from ideologically opposed sources. In contrast, the results based on web-browsing data showed that people with more extreme media diets are persuaded primarily by outlets with which they already agree.

"Together, these results suggest that inferences about media polarization may depend heavily on how individuals' media preferences are measured," the authors state in the paper.

"Our results affirm the value of harnessing real-world data to study political media," adds de Benedictis-Kessner. "Precise measurement of people's behavior in online news environments is difficult, but it is important to confront these measurement challenges due to the different conclusions that can arise about the dangers of political polarization."

Extending the research

As the scholars acknowledge, there are necessarily some questions left open by their work. For one, the current study focused on providing media content related to education policy, including issues such as school choice and charter schools. While education is a prominent issue for many citizens, it is an area that tends not to display as much polarization as some other topics in American life. It is possible that studies involving other political issues might reveal different dynamics.

"An interesting extension for this work would be to look at different issue areas, some of which might be more polarized than education," Wittenberg says.

She adds, "I hope the field can move toward testing a broader array of measures to see how they cohere, and I think there's going to be a lot of interesting and actionable insights. Our goal is not to say, 'Here is a perfect measure you should go out and use.' It's to nudge people to think about how they are measuring these preferences."

This story is republished courtesy of MIT News (web.mit.edu/newsoffice/), a popular site that covers news about MIT research, innovation and teaching.