Crowdsourcing creates a database of surfaces

August 28, 2013 by Bill Steele, Cornell University
Crowdsourcing creates a database of surfaces
Crowdsourcing builds a database. Two thousand online workers selected and annotated examples of materials from Flickr photos. Credit: Bala Group

( —Computer graphics are moving off the movie screen and into everyday life. Home remodeling specialists, for example, may soon be able to to show you how your kitchen would look with marble countertops or stainless steel appliances. To do this, computers have to be able to recognize and simulate common materials; so Cornell researchers have drawn on uniquely human skills to build a database of surfaces computers can work with.

OpenSurfaces offers more than 25,000 annotated images that may be used by architects, designers and home remodelers in visualizing their work, and could be a rich source for and computer vision researchers looking for ways to recognize materials or synthesize images of them. Of particular value, the researchers said, is that the images were collected from the real world – the Flickr photo-sharing site – rather than sterile laboratory samples. "This catches real materials that show up in the world, including wear and tear and weathering," said Kavita Bala, associate professor of computer science. "One of the things missing in computer graphics is the realism of normal life."

The crowdsourcing user interface the researchers developed to teach a team of human reviewers to find and describe the images also represents a valuable contribution to the field. "People are very good at recognizing materials but very bad at communicating the information," Bala said.

Bala, along with Noah Snavely, assistant professor of computer science, and graduate students Sean Bell and Paul Upchurch, described their work at the 2013 SIGGRAPH conference, July 21-25 in Anaheim, Calif.

"First I want to understand how human beings perceive materials," Bala explained. "Then I want to create that recognize materials, and computer graphics algorithms that can produce images of those materials that have all the subtle features."

The researchers began by collecting about 100,000 images from Flickr. With home remodeling applications in mind, they looked for tags like "kitchen," "bedroom" or "living room." Then they turned to the Amazon Mechanical Turk (MTurk) service, which enables employers to hire online workers to perform tasks that computers are unable to do. They eventually built a workforce of about 2,000 people all over the world to select surfaces displayed in the photos, identify the material in the selection and add comments on the context and how the surface reflected light.

Getting the user interface right was the big challenge, Bala reported. "Given language barriers and cultural differences, you pretty much have to bulletproof the task," she said. "Drywall," for instance, describes many walls in the United States, but workers in other countries might enter "concrete" or "plaster." Eventually the researchers decided to let workers select from a drop-down list rather than entering free-form descriptions. Other properties like shininess and roughness were reported using sliders.

"Using perceptual language works better than graphics terminology," Bala explained. "Before we refined the design we were spending a lot of money because we couldn't get good data." At first, she said, it was costing about $3 for each material added to the database, but eventually the cost got down to 10 cents each.

Now in the works, Bala said, is an application to modify images by changing one material into another. Instead of just looking at pictures in a catalog, take a photo of your own kitchen and see how it would look with new countertops. Down the line, she added, devices like Google Glass might draw on the database to identify materials in the field.

Explore further: Software arranges photo lighting after the shoot

Related Stories

Software arranges photo lighting after the shoot

August 22, 2013

What often separates professional photographers from amateurs is their mastery of lighting. Lighting can control what parts of an image draw your attention, or whether an object looks expensive or cheesy. And even for pros, ...

Perfecting digital imaging (w/ Video)

July 23, 2013

Computer graphics and digital video lag behind reality; despite advances, the best software and video cameras still cannot seem to get computer-generated images and digital film to look exactly the way our eyes expect them ...

Computer program can identify sketches

September 13, 2012

(—Computers are good at speed, numbers, and massive amounts of data, but understanding the content of a simple drawing is more difficult. Researchers at Brown and the Technical University of Berlin have produced ...

Recommended for you

Augmented reality takes 3-D printing to next level

February 20, 2018

Cornell researchers are taking 3-D printing and 3-D modeling to a new level by using augmented reality (AR) to allow designers to design in physical space while a robotic arm rapidly prints the work.

What do you get when you cross an airplane with a submarine?

February 15, 2018

Researchers from North Carolina State University have developed the first unmanned, fixed-wing aircraft that is capable of traveling both through the air and under the water – transitioning repeatedly between sky and sea. ...


Adjust slider to filter visible comments by rank

Display comments: newest first

not rated yet Aug 28, 2013
Home remodeling specialists, for example, may soon be able to to show you how your kitchen would look with marble countertops or stainless steel appliances
... take a photo of your own kitchen and see how it would look with new countertops

Erm. I'm currently having my kitchen redone. The friggin' guys at our local hardware store could already do that for me (took pictures, the freelance architect that set up shop there modeled it, including augmentations/alterations, for me in an hour and then we could click through various materials/textures/alternate configurations/fixtures. Didn't charge me a dime for it, either).
This is not future tech. This is old hat.

"This catches real materials that show up in the world, including wear and tear and weathering,"

Which isn't what I want to see. When I want something done I'm rather interested how it looks when new. That's sort of the whole point.

Might be a cool database for computer games, though.
not rated yet Sep 03, 2013
to create computer vision algorithms that recognize materials

I think that is the significant part of the story, maybe glossed over too quickly and lost in the poor choice of example. Assuming a successful implementation, the computer vision algorithms would be able to recognize materials even after wear and tear and weathering - 'in the wild'. Having recognized a surface, or even a material from it's surface, a computer (robot) could then make inferences about context, use and properties.

And going back to that kitchen remodeling, maybe some sort of augmented reality version, the software recognizing existing surfaces (with wear and tear) and replacing them on the fly in the new view with shiny new surfaces.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.