Robots learn to handle objects, understand places

September 2, 2011 By Bill Steele, Cornell University

After scanning a room, a robot points to the keyboard it was asked to locate. It uses context to identify objects, such as the fact that a keyboard is usually in front of a monitor.
( -- Infants spend their first few months learning to find their way around and manipulating objects, and they are very flexible about it: Cups can come in different shapes and sizes, but they all have handles. So do pitchers, so we pick them up the same way.

Similarly, your in the future will need the ability to generalize -- for example, to handle your particular set of dishes and put them in your particular dishwasher.

In Cornell's Personal Robotics Laboratory, a team led by Ashutosh Saxena, assistant professor of computer science, is teaching robots to manipulate objects and find their way around in new environments. They reported two examples of their work at the 2011 Robotics: Science and Systems Conference June 27 at the University of Southern California.

A common thread running through the research is "machine learning" -- programming a computer to observe events and find commonalities. With the right programming, for example, a computer can look at a wide array of cups, find their common characteristics and then be able to identify cups in the future. A similar process can teach a to find a cup's handle and grasp it correctly.

Other researchers have gone this far, but Saxena's team has found that placing objects is harder than picking them up, because there are many options. A cup is placed upright on a table, but upside down in a dishwasher, so the robot must be trained to make those decisions.

"We just show the robot some examples and it learns to generalize the placing strategies and applies them to objects that were not seen before," Saxena explained. "It learns about stability and other criteria for good placing for plates and cups, and when it sees a new object -- a bowl -- it applies them."

In early tests they placed a plate, mug, martini glass, bowl, candy cane, disc, spoon and on a flat surface, on a hook, in a stemware holder, in a pen holder and on several different dish racks.

Placing dishes in a rack is a challenging task for a robot. It must identify empty spaces and place the plate in the correct upright position.

Surveying its environment with a 3-D camera, the robot randomly tests small volumes of space as suitable locations for placement. For some objects it will test for "caging" -- the presence of vertical supports that would hold an upright. It also gives priority to "preferred" locations: A plate goes flat on a table, but upright in a dishwasher.

After training, their robot placed most objects correctly 98 percent of the time when it had seen the objects and environments previously, and 95 percent of the time when working with new objects in a new environment. Performance could be improved, the researchers suggested, by longer training.

But first, the robot has to find the dish rack.

Just as we unconsciously catalog the objects in a room when we walk in, Saxena and colleague Thorsten Joachims, associate professor of , have developed a system that enables a robot to scan a room and identify its objects. Pictures from the robot's 3-D camera are stitched together to form a 3-D image of an entire room that is then divided into segments, based on discontinuities and distances between objects. The goal is to label each segment.

The researchers trained a robot by giving it 24 office scenes and 28 home scenes in which they had labeled most objects. The computer examines such features as color, texture and what is nearby and decides what characteristics all objects with the same label have in common. In a new environment, it compares each segment of its scan with the objects in its memory and chooses the ones with the best fit.

"The novelty of this work is to learn the contextual relations in 3-D," Saxena said. "For identifying a keyboard it may be easier to locate the monitors first, because the keyboards are found below the monitors."

In tests, the robot correctly identified objects about 83 percent of the time in home scenes and 88 percent in offices. In a final test, it successfully located a keyboard in an unfamiliar room. Again, Saxena said, context gives this robot an advantage. The keyboard only shows up as a few pixels in the image, but the monitor is easily found, and the robot uses that information to locate the keyboard.

Robots still have a long way to go to learn like humans, the researchers admit. "I would be really happy if we could build a robot that would even act like a six-month-old baby," Saxena said.

Explore further: Robots could improve everyday life, do chores

Related Stories

Robots could improve everyday life, do chores

September 21, 2010

( -- They're mundane, yet daunting tasks: Tidying a messy room. Assembling a bookshelf from a kit of parts. Fetching a hairbrush for someone who can't do it herself. What if a robot could do it for you?

ARMAR-III, the robot that learns via touch (w/ Video)

November 17, 2010

( -- Researchers in Europe have created a robot that uses its body to learn how to think. It is able to learn how to interact with objects by touching them without needing to rely on a massive database of instructions ...

Care-O-bot 3: Always at your service

July 1, 2008

Who doesn’t long for household help at times? Service robots will soon be able to relieve us of heavy, dirty, monotonous or irksome tasks. Research scientists have now presented a new generation of household robots, the ...

Teaching robots to identify human activities

July 19, 2011

( -- If we someday live in "smart houses" or have personal robots to help around the home and office, they will need to be aware of what humans are doing. You don't remind grandpa to take his arthritis pills if ...

Robot, object, action!

October 29, 2010

Robotic demonstrators developed by European researchers produce compelling evidence that ‘thinking-by-doing’ is the machine cognition paradigm of the future. Robots act on objects and teach themselves in the process.

Robots learning from experience (w/ Video)

August 24, 2010

Software that enables robots to move objects about a room, building up ever-more knowledge about their environment, is an important step forward in artificial intelligence.

Recommended for you

After a reset, Сuriosity is operating normally

February 23, 2019

NASA's Curiosity rover is busy making new discoveries on Mars. The rover has been climbing Mount Sharp since 2014 and recently reached a clay region that may offer new clues about the ancient Martian environment's potential ...

Study: With Twitter, race of the messenger matters

February 23, 2019

When NFL player Colin Kaepernick took a knee during the national anthem to protest police brutality and racial injustice, the ensuing debate took traditional and social media by storm. University of Kansas researchers have ...

Researchers engineer a tougher fiber

February 22, 2019

North Carolina State University researchers have developed a fiber that combines the elasticity of rubber with the strength of a metal, resulting in a tougher material that could be incorporated into soft robotics, packaging ...


Adjust slider to filter visible comments by rank

Display comments: newest first

5 / 5 (1) Sep 02, 2011
well since robots have near limitless data storage, 24 scenes could be 24000000 scenes and web-linked learning between models passed around.
1 / 5 (1) Sep 02, 2011
Would have been nice to see a video of this robot in action.
not rated yet Sep 02, 2011
Similarly, your personal robot in the future will need the ability to generalize -- for example, to handle your particular set of dishes and put them in your particular dishwasher.
This is deduction, not induction. It's working from a general set of knowledge to learn specificity. Training mode would be inductive, and operating mode would be deductive.
not rated yet Sep 05, 2011
What they still have is just symbol recognition. They don't recognize the objects at all.

If you show the robot different kinds of chairs, it will assign the symbol "chair" to a few common geometric and visual properties of chairs. The context of the chair is just an extension of this process - chairs are found near tables.

But it doesn't get "chair". If you put a large chair next to a small table, it will get them wrong. The "chairness" or "tableness" of these objects doesn't depend on how they are, but what they're used for, and since the machine only understands symbols and their relations, it has no idea about why a chair is a chair when you sit on it, but a table when you put your coffee mug on it. You may teach it this relationship as well, but take away the coffee mug and it's clueless again.

This is why I think the AI researchers are going at the problem the wrong way. They're not making intelligence, they're just making something that acts like it through clever programmig.
5 / 5 (1) Sep 05, 2011
Would have been nice to see a video of this robot in action.

Have you seen the video on this page? I find it very impressive.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.