Scholars show new method of harvesting crowd wisdom

January 25, 2017 by Peter Dizikes, Massachusetts Institute of Technology
Credit: Massachusetts Institute of Technology

The wisdom of crowds is not always perfect. But two scholars at MIT's Sloan Neuroeconomics Lab, along with a colleague at Princeton University, have found a way to make it better.

Their method, explained in a newly published paper, uses a technique the researchers call the "surprisingly popular" algorithm to better extract correct answers from large groups of . As such, it could refine wisdom-of-crowds surveys, which are used in political and economic forecasting, as well as many other collective activities, from pricing artworks to grading scientific research proposals.

The new method is simple. For a given question, people are asked two things: What they think the right is, and what they think popular opinion will be. The variation between the two aggregate responses indicates the correct answer.

"In situations where there is enough information in the crowd to determine the correct answer to a question, that answer will be the one [that] most outperforms expectations," says paper co-author Drazen Prelec, a professor at the MIT Sloan School of Management as well as the Department of Economics and the Department of Brain and Cognitive Sciences.

The paper is built on both theoretical and empirical work. The researchers first derived their result mathematically, then assessed how it works in practice, through surveys spanning a range of subjects, including U.S. state capitols, general knowledge, medical diagnoses by dermatologists, and art auction estimates.

Across all these areas, the researchers found that the "surprisingly popular" algorithm reduced errors by 21.3 percent compared to simple majority votes, and by 24.2 percent compared to basic confidence-weighted votes (where people express how confident they are in their answers). And it reduced errors by 22.2 percent compared to another kind of confidence-weighted votes, those taking the answers with the highest average confidence levels.

The paper, "A solution to the single-question crowd wisdom problem," is being published today in Nature. The authors are Prelec; John McCoy, a doctoral student in the MIT Department of Brain and Cognitive Sciences; and H. Sebastian Seung, a professor of neuroscience and computer science at Princeton University and a former MIT faculty member. Prelec and McCoy are also researchers in the MIT Neuroeconomics Laboratory, where Prelec is the principal investigator.

A capital idea

To see how the algorithm works in practice, consider a case the researchers tested. A group of people were asked a yes-or-no question: Is Philadelphia the capital of Pennsylvania? They were also asked to predict the prevalence of "yes" votes.

Philadelphia is not the capital of Pennsylvania; the correct answer is Harrisburg. But most people believe Philadelphia is the capital because it is a "large, historically significant city." Moreover, the people who mistakenly thought Philadelphia is the state capital largely thought other people would answer the same way. So they predicted that a very high percentage of people would answer "yes."

Meanwhile, a certain number of respondents knew that Harrisburg is the correct answer. However, a large portion of those people also anticipated that many other people would incorrectly think the capital is Philadelphia. So the people who themselves answered "no" still expected a very high percentage of "yes" answers.

That means the answer to the two questions—Is Philadelphia the capital? Will other people think so?—diverged. Almost everyone expected other people to answer "yes." But the actual percentage of people who answered "yes" was significantly lower. For this reason, the "no" answer was the "surprisingly popular" one, since it deviated from expectations of what the answer would be.

And since the "surprisingly popular" answer differed in the "no" direction, that tells us the correct answer: No, Philadelphia is not the capital.

The same principle applies no matter which direction responses deviate from expectations. When people were asked if Columbia is the capital of South Carolina, the opposite happened: More people answered "yes," compared to their expectations of how many people would say "yes." So the surprisingly popular answer was, correctly: Yes, Columbia is the capital.

The wisdom of subsets of crowds

In this sense, the "surprisingly popular" principle is not simply derived from the wisdom of crowds. Instead, it uses the knowledge of a well-informed subgroup of people within the larger crowd as a diagnostically powerful tool that points to the right answer.

"A lot of crowd wisdom weights people equally," McCoy explains. "But some people have more specialized knowledge." And those people—if they have both correct information and a correct sense of public perception—make a big difference.

This is the case across scenarios that the researchers tested. Consider art. The researchers asked art professionals to guess the price range for different contemporary artworks. Individually, art experts selected price ranges that were typically too low, perhaps because selecting a lower range is a reasonable, safe answer for an artwork that the expert does not recognize. Collectively, this makes the majority opinion of an expert panel even more biased in the direction of low prices.

And this is where the "surprisingly popular" principle makes a difference, since it does not depend on an absolute majority of expert opinion. Instead, suppose a relatively small number of experts believe a piece sold for $100,000, while anticipating that most other people will think it sold for less. In that case, the evaluations of those experts will lead the "surprisingly popular" answer to be that the artwork was more expensive than most people thought.

"The argument in this paper, in a very rough sense, is that people who expect to be in the minority deserve some extra attention," Prelec says.

Recovering truth

The scholars recognize that the "surprisingly popular" algorithm is not theoretically foolproof in practice. It is at least conceivable that people could anticipate a "surprisingly popular" opinion and try to subvert it, although that would be very hard to execute. It is also the case, as they write in the Nature paper, that "These claims are theoretical and do not guarantee success in practice, as actual respondents will fall short of ideal."

Still, the researchers themselves hope their work will be tested in a variety of settings. In the paper they express confidence that the "surprisingly popular" principle will prove durable, asserting: "Such knowledge can be exploited to recover truth even when traditional voting methods fail."

Explore further: Is that your final answer? Study suggests method for improving individual decisions

More information: "A solution to the single-question crowd wisdom problem," Nature, nature.com/articles/doi:10.1038/nature21054

Related Stories

Older beats younger when it comes to correcting mistakes

October 28, 2015

Findings from a new study challenge the notion that older adults always lag behind their younger counterparts when it comes to learning new things. The study, published in Psychological Science, a journal of the Association ...

Recommended for you

Solving the mystery of an unusual medieval text

July 20, 2018

When historian Rowan Dorin first stepped onto the Stanford campus in early 2017, he made it a habit to visit Green Library every week to dig through its collection of medieval documents and objects.

6 comments

Adjust slider to filter visible comments by rank

Display comments: newest first

Milou
2 / 5 (1) Jan 25, 2017
This study will be challenged big time. If your are the looser "it is rigged" if your are the winner "it is alternate facts". Win, win situation for a disaster. I hope I am wrong.
Dug
3 / 5 (1) Jan 25, 2017
"Crowd wisdom" is what we are currently experiencing in the election of our current president. It is also the demonstrated and mathematical flaw inherent to true democracy. The voting population (or any crowd sourced information) can't achieve a better decision outcome and or informational gathering experience than their averaged intelligence (which will be very close to the average intelligence of the population as a whole) will allow.

Given that the recent election did not provide the popular vote outcome (barely a decision of the majority) and the early behavior of this president - we can assume that electoral college award of this presidency represented something less than the population's average intelligence in decision making. Unless you are sourcing information from a crowd of well above average intelligence individuals, your outcomes will be average or less.
Whydening Gyre
4.5 / 5 (2) Jan 25, 2017
Given that the recent election did not provide the popular vote outcome (barely a decision of the majority) and the early behavior of this president - we can assume that electoral college award of this presidency represented something less than the population's average intelligence in decision making. Unless you are sourcing information from a crowd of well above average intelligence individuals, your outcomes will be average or less.

There's always that dang curve...:-)
It does, however, provide an insight into crowd psychology...
antialias_physorg
5 / 5 (1) Jan 26, 2017
"Crowd wisdom" is what we are currently experiencing in the election of our current president.

Well, they do point out that they leverage the:
"knowledge of a well-informed subgroup of people within the larger crowd "

I think no one in their right mind could argue that the American public classifies as 'well informed'.
Whydening Gyre
not rated yet Jan 26, 2017
"Crowd wisdom" is what we are currently experiencing in the election of our current president.

Well, they do point out that they leverage the:
"knowledge of a well-informed subgroup of people within the larger crowd "

I think no one in their right mind could argue that the American public classifies as 'well informed'.

Actually, we're OVER informed (with oftentimes conflicting info) -
In order to foment confusion, indecision and mistrust...
One of the easiest ways to control a crowd...
EWH
not rated yet Feb 02, 2017
@ Dug: I think Trump is the epitome of "surprisingly popular" in the sense the researchers used the term. I also think their use scenario where you don't know anything about the voters is a bit artificial. A better algorithm would be to detect whether an unusual answer was more popular among those with a history of being right when most were wrong. That would be highly correlated with intelligence in all topics and with knowledge as well in most topics, since intelligence is a measure of the difficulty of questions one can be expected to answer (Rasch model), so "hard" questions are exactly those that the majority gets wrong and the minority with the needed ability get right, the ones with "surprisingly popular" wrong answers..

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.