Crowd mentality

September 21, 2011 By Heather Wuebker
“As a group, people are collectively more intelligent than you might think,” says professor Mark Steyvers. “We all possess knowledge about different things, some more than others. If we pool these pieces of knowledge together, we can get a pretty accurate look at the big picture.” Credit: / Francesco Santalucia

Do you know what the price of gas will be in six months? How about the extent of the expected U.S. troop drawdown in Afghanistan by year’s end? As an individual, predicting such things with any degree of confidence may seem the stuff of science fiction, but UC Irvine cognitive scientists say it’s possible.

“As a group, people are collectively more intelligent than you might think,” says professor Mark Steyvers. “We all possess knowledge about different things, some more than others. If we pool these pieces of knowledge together, we can get a pretty accurate look at the big picture.”

It’s called the “” effect, and researchers are hoping to get your help putting it to the test.

Steyvers and fellow UCI cognitive sciences professors Michael Lee and William Batchelder are part of a national team given a grant by the Intelligence Advanced Research Projects Activity to develop statistical models based on the crowds concept that can, with increasing precision, forecast the future.

Working with the UCI trio are scientists from the University of Maryland, the University of Michigan, The Ohio State University, Fordham University, Wake Forest University, Wichita State University and private company Applied Research Associates.

They constitute one of five national teams to be funded by IARPA and must meet yearly goals to continue receiving support from the government agency.

In July, the UCI/multi-university team launched its models via a software program called Forecasting Ace. Through volunteer participation, the program will collect individual opinions on the likelihood of certain events within a specified time frame.

Before providing input, contributors must complete a short questionnaire in which they rate their subject-matter expertise in areas including science and technology, business and the economy, politics and policy, military and security, and sports and health.

After the self-assessment, participants can choose which questions about the future they’d like to answer. The software then aggregates and analyzes the collected data using models of human decision making created by the UCI scientists.

“Finding wisdom in crowds is fundamentally a cognitive science problem, because it’s about how people acquire knowledge and make judgments,” says Lee. The concept, while relatively recent, is similar to cultural consensus theory, which Batchelder has been developing for 30 years.

The newer computer and cognitive aspects draw on the expertise of Lee and Steyvers, who have been perfecting these modeling methods at UCI over the past four years through numerous lab experiments and analyses of real-world behavioral data. The professors, along with two student researchers, won a best-paper prize for this work at the July conference of the Cognitive Science Society.

The qualitative and quantitative results from these various studies have helped them construct novel ways of identifying experts by combining self-reported expertise levels and behavioral responses in their models.

As the time frame for a forecasting challenge – such as the likelihood of federal legislators agreeing to raise the U.S. debt ceiling by Aug. 2 – draws near, the UCI team runs the collected data through the models. During this process, “ace analysts” – true experts in their field – are identified, and their opinions are appropriately weighted, helping researchers derive a relatively sound prediction of the upcoming event’s outcome.

They currently don’t know a prediction’s degree of accuracy until the event does or does not occur, but continued data aggregation lets them hone the models to account for more complex and precise forecasting.

The resulting applied project has implications for government agencies interested in improving the accuracy of intelligence analysis. It may also have applications in the fields of business, medicine and policy.

Funding for the endeavor began in May and runs through April 2012, at which time it may be extended. During that period, the research program will strive to achieve a 20 percent increase in forecast accuracy compared with state-of-the-art alternatives. Subsequent years will aim for growth of 35, 50 and more percentage points.

Explore further: National crowdsourcing project to better predict world events

Related Stories

Now you see him

March 8, 2011

Imagine if Harry Potter’s cloak were real , or that you could blot out the sight of something as easily as pressing a mute button to eliminate sound. To some, this seems like “pi in the sky” – fantastic ...

Consumer confidence hits five-year high in Michigan

October 27, 2010

( -- Despite Michigan’s continued economic malaise, residents’ optimism about the future is at its highest in nearly five years, according to Michigan State University’s latest State of the State ...

Recommended for you

Averaging the wisdom of crowds

December 12, 2017

The best decisions are made on the basis of the average of various estimates, as confirmed by the research of Dennie van Dolder and Martijn van den Assem, scientists at VU Amsterdam. Using data from Holland Casino promotional ...

1 comment

Adjust slider to filter visible comments by rank

Display comments: newest first

not rated yet Sep 21, 2011
The wisdom of crowds only works when other people do not know what others are thinking. Otherwise they start to align and make systematic mistakes en masse. See: Herd mentality or lemmings.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.