Knife-wielding robot trains for grocery checkout job using new coactive learning technique (w/ Video)

November 5, 2013, Cornell University

Cornell University engineers have taught a robot to work in a mock-supermarket checkout line, modifying a Baxter robot from Rethink Robotics in Boston to "coactively learn" from humans and make adjustments while an action is in progress.

"We give the a lot of flexibility in learning," said Ashutosh Saxena, assistant professor of computer science. "The robot can learn from corrective human feedback in order to plan its actions that are suitable to the environment and the objects present."

Saxena's research team will report their work at the Neural Information Processing Systems conference in Lake Tahoe, Calif., Dec. 5-8.

Modern industrial robots, like those on automobile assembly lines, have no brains, just memory. An operator programs the robot to move through the desired action; the robot can then repeat the exact same action every time an object goes by.

But off the assembly line, things get complicated: A working in a home has to handle tomatoes more gently than canned goods. If it needs to pick up and use a sharp kitchen knife, it should be smart enough to keep the blade away from humans.

The Baxter's arms have two elbows and a rotating wrist, so it's not always obvious to a human operator how best to move the arms to accomplish a particular task. So Saxena and graduate student Ashesh Jain drew on previous work, adding programming that lets the robot plan its own motions. It displays three possible trajectories on a where the operator can select the one that looks best.

Then humans can give corrective feedback. As the robot executes its movements, the operator can intervene, manually guiding the arms to fine-tune the . The robot has what the researchers call a "zero-G" mode, where the robot's arms hold their position against gravity but allow the operator to move them. The first correction may not be the best one, but it may be slightly better. The learning algorithm the researchers provided allows the robot to learn incrementally, refining its trajectory a little more each time the human operator makes adjustments or selects a trajectory on the touch screen. Even with weak but incrementally correct feedback from the user, the robot arrives at an optimal movement.

The robot learns to associate a particular trajectory with each type of object. A quick flip over might be the fastest way to move a cereal box, but that wouldn't work with a carton of eggs. Also, since eggs are fragile, the robot is taught that they shouldn't be lifted far above the counter. Likewise, the robot learns that sharp objects shouldn't be moved in a wide swing; they are held in close, away from people.

In tests with users who were not part of the research team, most users were able to train the robot successfully on a particular task with just five corrective feedbacks. The robots also were able to generalize what they learned, adjusting when the object, the environment or both were changed.

Explore further: Robots learn how to arrange objects by 'hallucinating' humans into their environment (w/ video)

More information: Early version of paper:

Related Stories

Amber 2 robot walks with a human gait (w/ Video)

October 25, 2013

( —The engineering team at Texas A&M's Amber robotics labs has been hard at work trying to improve one area of robotics that others seem to be ignoring—getting a robot to mimic the natural gait of a human being. ...

Gravity and the robot satellite attitude problem

November 4, 2013

Using an in-orbit robot to capturing a malfunctioning satellite that is tumbling out of control is currently just a theoretical idea. However, research inspired by nature to be published in the forthcoming issue of International ...

Recommended for you

Printing microelectrode array sensors on gummi candy

June 22, 2018

Microelectrodes can be used for direct measurement of electrical signals in the brain or heart. These applications require soft materials, however. With existing methods, attaching electrodes to such materials poses significant ...

EU copyright law passes key hurdle

June 20, 2018

A highly disputed European copyright law that could force online platforms such as Google and Facebook to pay for links to news content passed a key hurdle in the European Parliament on Wednesday.


Adjust slider to filter visible comments by rank

Display comments: newest first

not rated yet Nov 05, 2013
thats great and all but i wouldn't want to be the test driver of knife wielding robots.
1 / 5 (4) Nov 05, 2013
My grocery store has already eliminated half of the cashier jobs by installing self-checkout lanes. Now they'll be able to replace most of the remaining cashiers with robots.

So long as we have food stamps the grocery chain should do fine.
1 / 5 (6) Nov 05, 2013
My opinion: They won't replace check outs with robots, it would be cost prohibitive.

My prediction: Everything will have an IR, WiFi tag, all the items in your trolley/bag will be scanned automatically, and then you will either swipe/tap/automatically (using same process as your scanned groceries) pay for your groceries.

This will apply to all "mass" generic stores, with human beings only being used to restock shelves.

The tech already exists to do this, the only problems: human beings who traditionally held these jobs, human beings who don't trust an automatic scanner to correctly assess the contents of their trolley/account.

Which can be addressed relatively simply - your personal phone/watch/device will display an incremental calculator as you add each item to your trolley, and whatever computer regulates the door/scanner will confirm the total, check your device, reciept sent directly to your email (future: appropriate accounting software reviews).
1 / 5 (3) Nov 05, 2013
@Zera - I hope you're right that robots will be too cost prohibitive to replace human check-outs.

Also, robots might soon be able to restock shelves (Amazon is investing heavily in this technology) and I suspect that robots will eventually be able to do this cheaper and faster.
1 / 5 (5) Nov 06, 2013
@Shakescne21 - it is illogical to fear robots, one might as well fear ourself - it is the conditions of the mind we have the potential to impart on them we need to fear (predation, greed, envy). Qantas already has automated stacker/reclaimer tech in use, it's just adaptive tech we need to figure out - that being environments that don't follow linear 1,2,3 and then 4 logic.
not rated yet Nov 18, 2013
@Shakescene21, I think the technology already exists at a not as complex level. I think of it as a really really fancy tape drive arm. I think the main issue would be lining the product up in a place where the arm would be able to successfully grab an item and place it in it's designated place. Then, when the item is checked out, maybe using something like @ZERA is saying,then automagically decrements a counter out of an SAP type system that tells some system how many items to replace. It could probably be done seemlessly too so the consumer never sees it. For example back load the shelves so the last one in is the last one out. Now I'm thinking of some sort of bank like air tube.. Brain is turning now! Redesign supermarket aisles/buildings! AI/Robot influenced architecture! I'm not sure how you would handle the overhead to flip a profit but logically, I definitely think it's possible. If any of that made sense...

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.