Helping robots learn to see in 3-D

July 14, 2017 by Robin A. Smith
When fed 3-D models of household items in bird's-eye view (left), a new algorithm is able to guess what the objects are, and what their overall 3-D shapes should be. This image shows the guess in the center, and the actual 3-D model on the right. Credit: Ben Burchfiel

Autonomous robots can inspect nuclear power plants, clean up oil spills in the ocean, accompany fighter planes into combat and explore the surface of Mars.

Yet for all their talents, robots still can't make a cup of tea.

That's because tasks such as turning the stove on, fetching the kettle and finding the milk and sugar require perceptual abilities that, for most machines, are still a fantasy.

Among them is the ability to make sense of 3-D objects. While it's relatively straightforward for robots to "see" objects with cameras and other sensors, interpreting what they see, from a single glimpse, is more difficult.

Duke University graduate student Ben Burchfiel says the most sophisticated robots in the world can't yet do what most children do automatically, but he and his colleagues may be closer to a solution.

Burchfiel and his thesis advisor George Konidaris, now an assistant professor of computer science at Brown University, have developed new technology that enables machines to make sense of 3-D objects in a richer and more human-like way.

A robot that clears dishes off a table, for example, must be able to adapt to an enormous variety of bowls, platters and plates in different sizes and shapes, left in disarray on a cluttered surface.

Humans can glance at a new and intuitively know what it is, whether it is right side up, upside down or sideways, in full view or partially obscured by other objects.

Even when an object is partially hidden, we mentally fill in the parts we can't see.

Their robot perception algorithm can simultaneously guess what a new object is, and how it's oriented, without examining it from multiple angles first. It can also "imagine" any parts that are out of view.

A robot with this technology wouldn't need to see every side of a teapot, for example, to know that it probably has a handle, a lid and a spout, and whether it is sitting upright or off-kilter on the stove.

The researchers say their approach, which they presented July 12 at the 2017 Robotics: Science and Systems Conference in Cambridge, Massachusetts, makes fewer mistakes and is three times faster than the best current methods.

This is an important step toward robots that function alongside humans in homes and other real-world settings, which are less orderly and predictable than the highly controlled environment of the lab or the factory floor, Burchfiel said.

With their framework, the robot is given a limited number of training examples, and uses them to generalize to new objects.

"It's impractical to assume a robot has a detailed 3-D model of every possible object it might encounter, in advance," Burchfiel said.

The researchers trained their algorithm on a dataset of roughly 4,000 complete 3-D scans of common household objects: an assortment of bathtubs, beds, chairs, desks, dressers, monitors, nightstands, sofas, tables and toilets.

Each 3-D scan was converted into tens of thousands of little cubes, or voxels, stacked on top of each other like LEGO blocks to make them easier to process.

The algorithm learned categories of objects by combing through examples of each one and figuring out how they vary and how they stay the same, using a version of a technique called probabilistic .

When a spots something new—say, a bunk bed—it doesn't have to sift through its entire mental catalogue for a match. It learns, from prior examples, what characteristics beds tend to have.

Based on that prior knowledge, it has the power to generalize like a person would—to understand that two objects may be different, yet share properties that make them both a particular type of furniture.

To test the approach, the researchers fed the algorithm 908 new 3-D examples of the same 10 kinds of household items, viewed from the top.

From this single vantage point, the algorithm correctly guessed what most objects were, and what their overall 3-D shapes should be, including the concealed parts, about 75 percent of the time—compared with just over 50 percent for the state-of-the-art alternative.

It was also capable of recognizing objects that were rotated in various ways, which the best competing approaches can't do.

While the system is reasonably fast—the whole process takes about a second—it is still a far cry from human vision, Burchfiel said.

For one, both their algorithm and previous methods were easily fooled by objects that, from certain perspectives, look similar in shape. They might see a table from above, and mistake it for a dresser.

"Overall, we make a mistake a little less than 25 percent of the time, and the best alternative makes a mistake almost half the time, so it is a big improvement," Burchfiel said. "But it still isn't ready to move into your house. You don't want it putting a pillow in the dishwasher."

Now the team is working on scaling up their approach to enable robots to distinguish between thousands of types of objects at a time.

"Researchers have been teaching robots to recognize 3-D objects for a while now," Burchfield said. What's new, he explained, is the ability to both recognize something and fill in the blind spots in its field of vision, to reconstruct the parts it can't see.

"That has the potential to be invaluable in a lot of robotic applications," Burchfiel said.

Explore further: Robot uses social feedback to fetch objects intelligently

More information: "Bayesian Eigenobjects: A Unified Framework for 3D Robot Perception," Benjamin Burchfiel and George Konidaris. RSS 2017, July 12-16, 2017, Cambridge, Massachusetts.

Related Stories

Robot uses social feedback to fetch objects intelligently

March 6, 2017

If someone asks you to hand them a wrench from a table full of different sized wrenches, you'd probably pause and ask, "which one?" Robotics researchers from Brown University have now developed an algorithm that lets robots ...

New system learns how to grasp objects

June 9, 2017

Researchers at Bielefeld University have developed a grasp system with robot hands that autonomously familiarizes itself with novel objects. The new system works without previously knowing the characteristics of objects, ...

Meet the most nimble-fingered robot ever built

May 29, 2017

Grabbing the awkwardly shaped items that people pick up in their day-to-day lives is a slippery task for robots. Irregularly shaped items such as shoes, spray bottles, open boxes, even rubber duckies are easy for people to ...

Robot Boris learning to load a dishwasher (w/ Video)

September 12, 2014

Researchers at the University of Birmingham in the U.K. have set themselves an ambitious goal: programming a robot in such a way as to allow it to collect dishes, cutlery, etc. from a dinner table, and put it in a dishwasher. ...

Recommended for you

Google, EU dig in for long war

July 20, 2017

Google and the EU are gearing up for a battle that could last years, with the Silicon Valley behemoth facing a relentless challenge to its ambition to expand beyond search results.

Strengthening 3-D printed parts for real-world use

July 20, 2017

From aerospace and defense to digital dentistry and medical devices, 3-D printed parts are used in a variety of industries. Currently, 3-D printed parts are very fragile and only used in the prototyping phase of materials ...

Swimming robot probes Fukushima reactor to find melted fuel

July 19, 2017

An underwater robot entered a badly damaged reactor at Japan's crippled Fukushima nuclear plant Wednesday, capturing images of the harsh impact of its meltdown, including key structures that were torn and knocked out of place.

Microsoft cloud to help Baidu self-driving car effort

July 19, 2017

Microsoft's cloud computing platform will be used outside China for collaboration by members of a self-driving car alliance formed by Chinese internet search giant Baidu, the companies announced on Tuesday.

Making lab equipment on the cheap

July 18, 2017

Laboratory equipment is one of the largest cost factors in neuroscience. However, many experiments can be performed with good results using self-assembled setups involving 3-D printed components and self-programmed electronics. ...

3 comments

Adjust slider to filter visible comments by rank

Display comments: newest first

luke_w_bradley
not rated yet Jul 14, 2017
Super exciting, that's been the missing part of CV for a long time. It also makes clear vision is rooted in learning: A human guesses its a dresser, because there's a bed in the room, its not next to the bed, and a sock is lying on the floor nearby. I also just read about ghost imaging, after hearing about it on an army podcast, and I'm convinced it can unambiguously image in 3D if nothing else can, so we're basically there.
EmceeSquared
not rated yet Jul 15, 2017
Google's Tango sensors give mobile devices 3D vision (2D color, depth and motion) calibrated to an "Inertial Measurement Unit" (like the human inner ear). Giving the 3D space it's in (to the millimeter), its precise orientation in that space (to the hundredth of a degree). A "Visual Positioning System", recognizing indoor spaces (even if only seen by other devices before) and locations in them where GPS doesn't work. The reference frame for inserting Augmented Reality models too. So it will be in most phones within a few years.

So Google is crowdsourcing models of places all over the world, from billions of users. And tracking interactions within them, whether annotated in apps or just associated with other user actions.

All that big data is fed into Google's AI Tensorflow processing units, where Cloud processing as cheap as on a Pentium but instead as fast as the #7 "Top 500" supercomputer is machine learning what all that is. Millions of them.

AI is now.
zave
not rated yet Jul 16, 2017
i believe that Ben Burchfiel technology is better than what google is making for robot to see and learn with.
Because the robot learns to see with a algorithm inside the robot as oppose to relying on computers outside
of it.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.