Crowd-talk yields great answers, says university team

Sep 10, 2012 by Nancy Owano report
The Chorus system. User requests are forwarded to crowd workers, who then submit and vote on responses. Once sufficient agreement is reached, responses are made visible to users. The crowd’s working memory is updated by workers selecting lines from the conversation or summarizing important facts. Credit: Walter S. Lasecki et al.

(—Move over, Siri. Some researchers from the University of Rochester in collaboration with a University of California, Berkeley, mathematician/crowdsourcing entrepreneur, have come up with a killer personal assistant approach. "We introduce Chorus, a system that enables realtime, two-way natural language conversation between an end user and a crowd acting as a single agent." So begins their paper, "Speaking with the Crowd," suggesting the ideal artificial chat partner is the partner that is actually the work of contributions from many crowdsourced workers. The researchers propose a crowd-powered chat system that behaves as an online collaborative interface. They believe it one-ups existing systems because it can take on more complex tasks.

Walter S. Lasecki, Rachel Wesley, and Jeffrey P. Bigham from the University of Rochester worked with Anand Kulkarni, the cofounder of the crowdsourcing company MobileWorks, to create Chorus. They sought to demonstrate that the power of crowdsourcing might be able to go beyond simple tasks into the complex. "What we're really interested in is when a as a collective can do better than even a high-quality individual," said co-author Bigham.

How the Chorus system works: People talk to Chorus with an instant messaging window. User requests are forwarded to crowd workers, who submit and vote on responses. When agreement is reached, responses are made visible to users. The crowd's is updated by workers selecting lines from the conversation or summarizing important facts. According to the co-authors of the paper, "Chorus is capable of maintaining a consistent, on-topic conversation with end users across multiple sessions, despite individuals perpetually joining and leaving the crowd. This is enabled by using a curated shared dialogue history."

Put to the test, the researchers found that Chorus was able to keep up consistent conversations between a single user and of crowd participants. They said the talk was kept on focus. Also, Chorus was capable of retaining meaningful long-term conversational memory across multiple sessions, even as individual users changed. As for accuracy, workers were able to answer 84.6 percent of user queries correctly.

In all, say the authors, "These findings suggest that Chorus is a robust interface for allowing disparate members of the crowd to represent a single individual during natural language conversations as an alternative to software agents."

How robust these systems become as technologies evolve remains to be seen. What is known is that personal assistants such as Apple's Siri are very useful but do not come up to par with the conversational skills of a real person. As the researchers note, robust two-way conversations with software agents remain a challenge. Existing dialogue-based systems generally rely on a fixed input vocabularies or restricted phrasings, have a limited memory of past interactions, and use a fixed output vocabulary.

As for where Chorus can fit in the real world, the authors say that "In the future, we expect will have utility as a conversational partner and as a natural-language dialogue interface to existing systems."

Explore further: Computer program to take on world's best in Texas Hold 'em

More information: Research paper: Speaking With the Crowd,

Related Stories

iPhone app uses crowd sourcing to help the blind

May 13, 2011

( -- A team of innovative thinkers from several universities working together on a joint project to merge existing technology (the iPhone) with a real-world crowd sourcing application to help blind ...

Workforce from the digital cloud

Feb 29, 2012

By means of cloud computing, enterprises can access scalable computing power and storage capacity. A people cloud, by contrast, supplies a scalable number of workers via the internet. It is used when non-automated tasks are ...

Ubisoft Jams with Nintendo DS

Mar 17, 2007

The South by Southwest Festival (SXSW) isn't necessarily known for the product-announcement frenzy that occurs at CES, so it was a pleasant surprise to see a cool new toy on the trade show floor Thursday evening.

Crowdsourcing nutrition in a snap

Nov 01, 2011

Americans spend upwards of $40 billion a year on dieting advice and self-help books, but the first step in any healthy eating strategy is basic awareness -- what's on the plate.

Recommended for you

Ears, grips and fists take on mobile phone user ID

13 hours ago

A research project has been under way to explore a biometric authentication system dubbed Bodyprint, with interesting test results. Bodyprint has been designed to detect users' biometric features using the ...

Russian hackers read Obama emails, report says

Apr 25, 2015

Emails to and from President Barack Obama were read by Russian hackers last year in a breach of the White House's unclassified computer system, The New York Times said Saturday.

Supermarkets welcome cold-comfort edge of F1 aerofoils

Apr 25, 2015

UK-based Williams Advanced Engineering, the technology and engineering services business of the Williams Group, has collaborated with UK-based Aerofoil Energy to develop an aerodynamic device that can reduce ...

User comments : 7

Adjust slider to filter visible comments by rank

Display comments: newest first

5 / 5 (4) Sep 10, 2012
The problem with crowd sourcing is information sharing. People are overwhelmingly imitators versus innovators so if information is shared most people begin to follow the popular opinion. Consider CNN et al and public decisions such as whether or not to go to war. If the information is all public the most common answer is followed.

The same is true with the famous guess the number of jelly beans in the jar contest. If all the votes are kept hidden the correct answer appears. On the other hand if the votes are displayed in real-time (like the stock market or real-estate prices) the most common answer is followed.

Any such crowd sourcing system needs information to be temporarily hidden to force individuals to "think for themselves", revealing the consensus after the fact.
5 / 5 (1) Sep 11, 2012
What induces members of the crowd to participate?
How well does Chorus perform given controversial topics, or topics that large numbers of participants find abhorrent?
5 / 5 (1) Sep 11, 2012
Why should there automatically be no bias in crowd derived estimates. Is there any proof??
not rated yet Sep 11, 2012
I am reminded of the popular tune "50 Million Commies Can't Be Wrong," by Sherman and Larsen.
The robustness of the scientific method suggests 'consensus' decisions are not as reliable as empirical testing.
not rated yet Sep 11, 2012
While the crowd got 85% correct (higher than my experience - except in extremely pre-qualified groups), a single individual could at least theoretically have gotten 100. Therein lies the limitation of crowds they will always be averaging their answers and always be asymptotic to 100%. Ultimately, this is computer augmented decision by committee. Somehow I can't but not think of how efficient Congress is - when I think of crowd sourcing decisions.
not rated yet Sep 12, 2012
Why should there automatically be no bias in crowd derived estimates. Is there any proof??

Crowdsourcing is not a new concept. It has been "played-around" with for years. Usually in the Social Sciences departments

My impression from what I remember, is that the most extreme outliers (biased opinions) will mostly cancel each other. Within a given range about a possible "average" there is always the possibility of some bias, which may not necessarily be a bad thing. It all depends on your definition of "bias."
not rated yet Sep 18, 2012
a single individual could at least theoretically have gotten 100. Therein lies the limitation of crowds they will always be averaging their answers and always be asymptotic to 100%.

It's not clear what 84.6% accuracy means without comparing to the average and best/worst accuracy of the individuals in the same pool. Also, a crowd can theoretically get 100% in the same scenario where any individual gets 100%. Just give that person 100% of the influence. (You'd have to identify that person a priori in either case.)

There is nothing inherent in the concept that requires "averaging" of answers in any particular way. You could have a reputation system to identify the strengths/weaknesses of active crowd members and adjust the weight of their opinions, use NLP to group questions with likely subject matter experts, try out different voting schemes, combine an ensemble of small crowds instead of one big crowd, etc.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.