Need an expert? Try the crowd

August 14, 2012 By Joshua E. Brown, University of Vermont
Can a crowd be an expert? Two UVM scientists think the answer is yes. (photo: James Cridland)

In 1714, the British government held a contest. They offered a large cash prize to anyone who could solve the vexing "longitude problem" — how to determine a ship's east/west position on the open ocean — since none of their naval experts had been able to do so.

Lots of people gave it a try. One of them, a self-educated carpenter named John Harrison, invented the marine chronometer — a rugged and highly precise clock — that did the trick. For the first time, sailors could accurately determine their location at sea.

A centuries-old problem was solved. And, arguably, was born.

Crowdsourcing is basically what it sounds like: posing a question or asking for help from a large group of people. Coined as a term in 2006, crowdsourcing has taken off in the internet era. Think of Wikipedia, and its thousands of unpaid contributors, now vastly larger than the Encyclopedia Britannica.

Crowdsourcing has allowed many problems to be solved that would be impossible for experts alone. Astronomers rely on an army of volunteers to scan for new galaxies. At, citizens have linked their home computers to yield more than a hundred million hours of climate modeling; it's the world's largest forecasting experiment.

But what if experts didn't simply ask the crowd to donate time or answer questions? What if the crowd was asked to decide what questions to ask in the first place?

Could the crowd itself be the expert?

That's what a team at the University of Vermont decided to explore — and the answer seems to be yes.

Prediction from the people

Josh Bongard and Paul Hines, professors in UVM's College of Engineering and Mathematical Sciences, and their students, set out to discover if volunteers who visited two different websites could pose, refine, and answer questions of each other — that could effectively predict the volunteers' body weight and home electricity use.

The experiment, the first of its kind, was a success: the self-directed questions and answers by visitors to the websites led to computer models that effectively predict user's monthly electricity consumption and body mass index.

Their results, "Crowdsourcing Predictors of Behavioral Outcomes," were published in a recent edition of IEEE Transactions: Systems, Man and Cybernetics, a journal of the Institute of Electrical and Electronics Engineers.

"It's proof of concept that a crowd actually can come up with good questions that lead to good hypotheses," says Bongard, an expert on machine science.

In other words, the wisdom of the can be harnessed to determine which variables to study, the UVM project shows — and at the same time provide a pool of data by responding to the questions they ask of each other.

"The result is a crowdsourced predictive model," the Vermont scientists write.

Unexpected angles

Some of the questions the volunteers posed were obvious. For example, on the website dedicated to exploring body weight, visitors came up with the question: "Do you think of yourself as overweight?" And, no surprise, that proved to be the question with the most power to predict people's body weight.

But some questions posed by the volunteers were less obvious. "We had some eye-openers," Bongard says. "How often do you masturbate a month?" might not be the first question asked by weight-loss experts, but it proved to be the second-most-predictive question of the volunteer's self-reported weights — more predictive than "how often do you eat during a day?"

"Sometimes the general public has intuition about stuff that experts miss — there's a long literature on this," Hines says.

"It's those people who are very underweight or very overweight who might have an explanation for why they're at these extremes — and some of those explanations might not be a simple combination of diet and exercise," says Bongard. "There might be other things that experts missed."

Cause and correlation

The researchers are quick to note that the variables revealed by the evolving Q&A on the experimental websites are simply correlated to outcomes — body weight and electricity use — not necessarily the cause.

"We're not arguing that this study is actually predictive of the causes," says Hines, "but improvements to this method may lead in that direction."

Nor do the scientists make claim to being experts on body weight or to be providing recommendations on health or diet (though Hines is an expert on electricity, and the EnergyMinder site he and his students developed for this project has a larger aim to help citizens understand and reduce their household energy use.)

"We're simply investigating the question: could you involve participants in the hypothesis-generation part of the scientific process?" Bongard says. "Our paper is a demonstration of this methodology."

"Going forward, this approach may allow us to involve the public in deciding what it is that is interesting to study," says Hines. "It's potentially a new way to do science."

And there are many reasons why this new approach might be helpful. In addition to forces that experts might simply not know about — "can we elicit unexpected predictors that an expert would not have come up with sitting in his office?" Hines asks — experts often have deeply held biases.

Faster discoveries

But the UVM team primarily sees their new approach as potentially helping to accelerate the process of scientific discovery. The need for expert involvement — in shaping, say, what questions to ask on a survey or what variable to change to optimize an engineering design — "can become a bottleneck to new insights," the scientists write.

"We're looking for an experimental platform where, instead of waiting to read a journal article every year about what's been learned about obesity," Bongard says, "a research site could be changing and updating new findings constantly as people add their questions and insights."

The goal: "exponential rises," the UVM scientists write, in the discovery of what causes behaviors and patterns — probably driven by the people who care about them the most. For example, "it might be smokers or people suffering from various diseases," says Bongard. The team thinks this new approach to science could "mirror the exponential growth found in other online collaborative communities," they write.

"We're all problem-solving animals," says Bongard, "so can we exploit that? Instead of just exploiting the cycles of your computer or your ability to say 'yes' or 'no' on a survey — can we exploit your creative brain?"

Explore further: For robust robots, let them be babies first (w/ Video)

Related Stories

10-year-old problem in theoretical computer science falls

July 30, 2012

( -- Interactive proofs, which MIT researchers helped pioneer, have emerged as one of the major research topics in theoretical computer science. In the classic interactive proof, a questioner with limited computational ...

Being in the 'no': questions influence what we remember

September 14, 2011

Imagine that you are sitting in the park, deeply engaged in a conversation with your loved one. A group of teenagers pass by in front of you. The next day you learn that the police are looking for someone to identify them ...

Recommended for you

Cryptocurrency rivals snap at Bitcoin's heels

January 14, 2018

Bitcoin may be the most famous cryptocurrency but, despite a dizzying rise, it's not the most lucrative one and far from alone in a universe that counts 1,400 rivals, and counting.

Top takeaways from Consumers Electronics Show

January 13, 2018

The 2018 Consumer Electronics Show, which concluded Friday in Las Vegas, drew some 4,000 exhibitors from dozens of countries and more than 170,000 attendees, showcased some of the latest from the technology world.

Finnish firm detects new Intel security flaw

January 12, 2018

A new security flaw has been found in Intel hardware which could enable hackers to access corporate laptops remotely, Finnish cybersecurity specialist F-Secure said on Friday.


Adjust slider to filter visible comments by rank

Display comments: newest first

1 / 5 (6) Aug 14, 2012
What really motivated the brittish government to have that contest? I mean, it's practically unheard of that a government asks the public for help with a problem, even though they never solve anything by themselves.
1 / 5 (5) Aug 14, 2012
The government is the people. This was some time ago as well.

We are a republic not a true democracy. In ancient Greece there was the predecessor to the modern politician, people would gather in forums and leaders in the discussions would emerge. But as it turns out in today society only the greediest and most cutthroat of leaders rise to the top.

Nowadays we have the means, ability and the desire to be a true democracy. There could be individual voting on every single issue. The federal govt. could finally get out of the states business because people could structure themselves.

I don't know if we are quite there yet.... Fix my road or your road first, right? Maybe if everyone learns to concede when it's not pertinent enough to them. The only way left seems to be a hellish nazi future where everything is documented and all the t's are crossed.

Or we could all just get naked and run around, what does the crowd have to say about that?
3 / 5 (2) Aug 15, 2012
The gov't is the people ? Bwahahahah!

Legally, the STATE and the gov't aren't even the same thing.

Nowadays what the Brits did will never work. No one would believe the guy unless he whipped out some paper credentials.
not rated yet Aug 15, 2012
Dava Sobel wrote about this, you have heard of the solution but there were some unconventional suggestions http://en.wikiped...sympathy
4.3 / 5 (9) Aug 15, 2012
In general, groups are smarter than individuals. But this is only true where the group has reasonably similar levels of knowledge.

I don't expect a random group to be able to design a bridge as well as an experienced engineer.

I don't expect a random group to be able to design a Particle Accelerator as effectively as trained physicists.

On the other hand, for social issues, there generally are no cut and dried optimal solutions. So group experience can generally be effectively relied upon to produce near optimal results.

But even here there are caveats.

The public must be smart enough to govern themselves.

Looking at the failure that is America, we see that the American failure is due to the dumbing and diming of the American people. Particularly American Conservatives.
1 / 5 (3) Aug 15, 2012
I'm not sure if I agree, I see your point but I think people need to have a broader width of education overall, and for a social model to work, it needs to come from people that have experience and knowledge of how society work, and how people relate to it, to each other and how they work with each other.
1.8 / 5 (5) Aug 15, 2012
You should always ask more experts at the same moment, because the experts are susceptible to professional bias the more, the more they're specialized in area of their interest (1, 2, 3, 4). At the moment, when the opinions of all experts agree mutually, then there is a reasonable possibility, their opinion is correct - until you don't ask the fundamental problem, which the opinion of all experts is based on. In my experience, at the case of such a problems the laymans tend to be as correct/false, as the professional experts.
1 / 5 (3) Aug 15, 2012
@Eoprime - could you explain, why did you downvote a post, which fits the meaning of actual article and which is supported with four another articles linked? Apparently you're one of "infallible experts" too.
5 / 5 (1) Aug 15, 2012
What "crowdsourcing" means is that: people who know the stuff are drawn to answer the question. This is different than "expert": which was paid to find the answer. -Alot of people really do know stuff they consider trivial that others do not.
not rated yet Aug 15, 2012
The Soviets practiced the same sort of crowd sourcing. They would add some problems that none of their scientists could solve into the annual academic olympiads. They had a perfect success rate. Feynman also discussed something similar when he trained some boy scouts, who had only high school math, to help with the calculations for the atomic bomb. Once they had a grasp of the problems their efficiency and contributions vastly improved.

The crowd doesn't need a fundamental understanding of a topic to actually arrive at the right answer. They just need a basic understanding and a direction and they'll handle the rest. It's pretty cool how that works. I'd love to see more work done on this topic.
3.4 / 5 (5) Aug 15, 2012
When rational people trust the opinions of experts in their field, then things usually go well. But when they start to hold opinions that are opposed to the knowledge held by experts and begin to deride those experts as "scientific twats", "Scientific Mafia", an unworthy "intellectual elite", or members of an anti-social "conspiracy of science" against them, then crowd sourcing fails.

We see these attacks on science almost exclusively from Libertarians, Republicans and other Conservatives in an ever growing number of fields of science. Archaeology, Biology - particularly evolutionary biology, physics - particularly relativity and cosmology - Climatology and of course the constant attacks by Conservatives on the field of Ecology.

1 / 5 (4) Aug 15, 2012
When Andrea Rossi will sell his megawatt-scale cold fusion device based on technology, which mainstream experts claimed impossible for twenty years, then I'll enjoy your opinion again...
2.3 / 5 (3) Aug 15, 2012
@Eoprime - could you explain, why did you downvote a post, which fits the meaning of actual article and which is supported with four another articles linked? Apparently you're one of "infallible experts" too.

Don't bother. There are lately a few names which have been registered in the last few weeks just for the purpose of down rating everything they don't like. They never post anything either, so you can't do the same to them. Be proud that your posts irked their moronic, imbecile, cowardly mentality that they had to go to such lousy method (oops! sorry, I really didn't mean that- it will be an utter insult to the whole louse family!)
5 / 5 (3) Aug 15, 2012
I look forward to the day when he sells one to someone other than himself.

Let us know when it happens.

"When Andrea Rossi will sell his megawatt-scale cold fusion device" - ValeriaT
1 / 5 (3) Aug 16, 2012
Let us know when it happens
Ignorants are ignoring.
The questions is, whether the important findings should be researched with mainstream physicists just when they get the megawatt scale? Are we really paying the physicists just for "basic" research without practical applications, or the research when everyone can be sure with its results?
Apparently for people like you it's more comfortable to wait for nuclear war for the rest of oil sources, because they feel sufficiently safe. The era, when Americans were scared of nuclear weapons is already gone.
1 / 5 (1) Aug 16, 2012
Haven't these people heard of Its been around since 1998.
1 / 5 (4) Aug 19, 2012
Some individuals were a crowd all by themselves... Leonardo, Einstein, David Bohm, Aristarchus, Democritus, Giordano Bruno, too many to mention, and most of them far ahead of their time and societal paradigm.
1 / 5 (2) Aug 19, 2012
Some individuals were a crowd all by themselves
This is just a twisting of democracy meaning. The crowds are anonymous by their very definition. BTW The crowds aren't synonym of non-conformal thinking of bright individuals - on the contrary. What we need is the balancing of the opinion of both experts, both the vox populi..
1 / 5 (2) Aug 19, 2012
a practical example of crowdsoursing: a retraction based on public review
5 / 5 (1) Aug 19, 2012
If that site had any worldly impact, they would have heard of it wouldn't they?

"Haven't these people heard of Its been around since 1998." = CapitalismHasFailed

So your question provides it's own answer.
1 / 5 (3) Aug 19, 2012
@Valeria - my metaphorical analogy was obviously beyond your linguistically cognitive capacities to interpret it in the manner intended - so, my apologies. Some will undoubtedly understand what was meant...
2.3 / 5 (3) Aug 19, 2012
Looking at the failure that is America, we see that the American failure is due to the dumbing and diming of the American people. Particularly American Conservatives.
There is some truth to this. When the rouble currency collapsed due to machinations by the Rothschilds' proxy Soros, Russian society did not fall apart simply because there was no money. Nothing happened at all! Contrast this to disasters in the USA. Inevitably there is looting, raping, arson and in New Orleans even reports of cannibalism. Moreover the police become authoritarian and go door to door raiding and looting themselves. USA is the worst place to be in the event of disasters, because there is no culture only a culture of hustling and opportunism. I'm sure it wasn't always that way.
Aug 20, 2012
This comment has been removed by a moderator.
1 / 5 (2) Sep 12, 2012
Companies like Innocentive are the holy grail of innovation crowd sourcing

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.