Making better use of the crowd

December 2, 2016

Over the last decade, computer scientists have harnessed crowds of Internet users to solve tasks that are notoriously difficult to crack with computers alone, such as determining whether an image contains a tree, rating the relevance of websites, and verifying phone numbers.

The machine learning community was early to embrace so-called crowdsourcing to quickly and inexpensively obtain the vast quantities of labeled needed to train machine learning systems how to classify images or recognize speech, for example. Labeled data are essentially sets of teaching examples, such as pictures of cats that are tagged with the word "cat."

Usually this handoff of labeled data is where interaction with the crowd ends. Are there better ways to make use of the crowd?

On December 5, I will showcase innovative uses of crowdsourcing that go beyond data collection during a crowdsourcing tutorial at NIPS, the premier international machine learning conference, held this year in Barcelona. The tutorial will also dive into recent research aimed at understanding who crowdworkers are and how they behave, which could inform best practices for interacting with the crowd.

Innovative uses of crowdsourcing that go beyond the collection of data include:

  • Harnessing the crowd to improve machine-learning models including extracting features from labeled data most relevant for model training and evaluation of learned models.
  • Leveraging the complementary strengths of humans and machines to achieve more than either can alone. Potential applications of these so-called hybrid-intelligence systems include real-time on-demand closed captioning of day-to-day conversations and crowd-powered writing and editing.
  • Using crowdsourcing platforms to recruit large pools of subjects for experiments designed to study the effects of human behavior when reasoning about the performance of computer systems, which could lead to better designed algorithms and systems.

Recent research—both qualitative and quantitative—has opened the black box of crowdsourcing to uncover that crowdworkers are not just independent contractors, but rather a network with a rich communication structure. Meanwhile, experiments have explored how to boost the quality of crowdwork using both well-designed monetary incentives (such as performance-based payments) and intrinsic motivation (such as piqued curiosity).

This research has much to teach us about how to most effectively interact with the . (Hint: Be respectful, be responsive, be clear.)

Crowdsourcing has the potential for major impact on the way we design and AI systems, but to unleash this potential we need more creative minds exploring novel ways to use it.

Explore further: Crowdsourcing provides remarkable benefits for companies

Related Stories

Crowdsourcing provides remarkable benefits for companies

August 16, 2016

According to a new study from the University of Vaasa "Generic Crowdsourcing Model for Holistic Innovation Management" organizations should be more customer focused in innovation production. The study indicates that crowdsourcing ...

Crowdsourcing—do the benefits outweigh the low-wage pay?

September 22, 2016

A millennial wakes up late for work. Usually she takes the bus, but she has an early meeting so she requests an Uber ride. After her meeting, she realizes she forgot to pick up a dessert for a dinner party tonight. It's a ...

Fighting crime through crowdsourcing

November 12, 2014

Crowdsourcing utilizes the input of a crowd of online users to collaboratively solve problems. To advance this emerging technology, researchers at the University of Miami are developing a computing model that uses crowdsourcing ...

Recommended for you

The wet road to fast and stable batteries

December 14, 2017

An international team of scientists—including several researchers from the U.S. Department of Energy's (DOE) Argonne National Laboratory—has discovered an anode battery material with superfast charging and stable operation ...

US faces moment of truth on 'net neutrality'

December 14, 2017

The acrimonious battle over "net neutrality" in America comes to a head Thursday with a US agency set to vote to roll back rules enacted two years earlier aimed at preventing a "two-speed" internet.


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.