Co-robots team up with humans

Charlie Kemp is giving robots common sense. And that's good news for Californian Henry Evans.

Ten years ago, Evans suffered a stroke that left him with limited mobility. Over the past two years, he's been working with Kemp, an associate professor in the Wallace H. Coulter Department of Biomedical Engineering at Georgia Tech and Emory University, to develop and test robots that help him shave, adjust a blanket when he's cold, and even scratch an annoying itch.

"We did things with the robots that I never could have imagined," said Evans, who contacted Kemp after seeing him on a CNN broadcast about health care robots.

Robots working directly with people – even helping them shave – is both challenging and unusual. Most robots today work in manufacturing facilities where, for safety reasons, they stay far away from humans. But Georgia Tech robotics researchers believe people and robots can accomplish much more by working together – as long as the robots have common sense to know, for instance, how much force humans apply when shaving.

"A major challenge for health care robots is that they lack so much of the knowledge and experience that people take for granted," said Kemp. "To us, it's just common sense that everybody has; for robots, it's a serious impediment."

Giving robots common sense is just one milestone on the path to the kinds of collaboration that will be required to meet the needs of a growing population of older persons. Beyond personal care, the benefits of co-robotics are many. To produce better products more efficiently, manufacturing robots will need to team up with humans, each contributing unique abilities. And in defense and homeland security, robots will increasingly have to take on the dangerous jobs, leveraging people's skills while protecting them from harm.

Robots as appliances

For co-robotics to become reality, robots will have to be easy enough for ordinary people to use.

"We can't make the assumption that potential home users of robots have even used an iPad," said Henrik Christensen, a professor in Georgia Tech's School of Interactive Computing and executive director of the Institute for Robotics and Intelligent Machines (IRIM). "Instead of looking at the technology and the technological capabilities of robots, we need to look at making this as simple as possible for the person at the other end. If we don't, people just won't use robots."

Christensen believes robots will one day be more like home appliances. Unpack them, turn them on, and start using them. Fortunately, computing power, sensing, vision, speech recognition, and other technologies have advanced in power and fallen in cost to the point where it'll be possible to produce inexpensive, but highly capable home robots.

"We are at a point where we can make technology that is fluent enough that people can use it," added Christensen, who holds the title of KUKA Chair of Robotics and Distinguished Professor. "We need to reach the level of fluency where getting a robot is like getting a new assistant in the home. It needs to be as easy as turning on the television."

Beyond ease of use, having robots work directly with people will require both human and machine to understand each other better. Robots will have to recognize what humans are trying to do so they can anticipate their needs – and avoid inadvertently harming them.

For example, a robot hauling laundry down a narrow hallway may sense that a human is approaching it. When two humans meet each other in a hallway, they know to turn their bodies to avoid running into each other. Today's robots would stop what they're doing until the human passes, but Christensen believes future robots can be given more human-like skills.

"This opens up some interesting issues for creating a dialogue in which a human and a robot can operate collaboratively," he added.

Mathematics tutors

As class sizes increase, teachers are busier than ever – and so are parents. So who's going to help kids understand critical math concepts?

Ayanna Howard believes socially aware robots may be part of the answer. Mathematics follows clear rules – rules that robots can understand. And because experienced teachers consistently see the same math misunderstandings, their "case studies" can be collected and given to robots so they can lead a child to the right solution.

More challenging is giving robots the ability to know when kids are bored or frustrated so the machine can provide the right kind of encouragement. "We look at things like eye gaze, interaction in terms of choosing the right answer, or taking too long to answer a specific problem," said Howard, who is the Motorola Foundation Professor in the Georgia Tech School of Electrical and Computer Engineering. "We have modeled the differences between frustration and boredom."

In this National Science Foundation-sponsored work, Howard is also using similar approaches for children who need therapy for disabilities such as cerebral palsy. Success of both efforts is further enhanced by the fascination kids have for robots.

"We're not substituting for therapists or teachers, but we're trying to imitate how a human interacts with children," she said. "We will always have teachers or clinicians involved. There will be a feedback loop from the robot to the clinician or teacher."

Howard's team is already testing two different robots for therapy and tutoring in Atlanta area schools, and has completed several of the modules they'll use, ranging from fifth-grade math to calculus. The researchers are using software to substitute for costly hardware in an effort to keep the devices below a specific price point: the cost of an Xbox game console.

"If you buy your kids an Xbox," she reasoned, "you should be able to buy them a robotic tutor."

Factory floor collaboration

Most robots today work with other robots on the factory floor, welding parts, painting or handling other repetitive jobs. Meanwhile, human workers handle tasks that are too complicated for robots or require significant judgment.

Aaron Bobick sees a potential partnership there.

In a project being done for BMW in South Carolina, Bobick is helping robots anticipate the needs of human workers building sub-assemblies for automotive consoles. Right now, humans must fetch the parts needed for each step, scanning them first to ensure that they are the right components for each vehicle before installing them.

"We have split the task up so the machine does the fetching and scanning, while the human does the inserting and installing," said Bobick, a professor in Georgia Tech's School of Interactive Computing. "The idea is for a fluid interaction – a fluid choreography, with the robot continually aware of what the human is doing."

Among the key issues in teaming robots with people: helping robots anticipate what people intend to do so the human-robot team can produce a better product more efficiently.

"The robots need to clearly understand what the human is doing – and be able to anticipate the next action," he said. "If the robot has to fetch the next part, the sooner it can know what part the human is going to need, the more in advance it can start, so that by the time the part is needed, the robot has it ready."

Safety is another obvious consideration. Like a good assistant, robots can be programmed to stop what they are doing or get out of the way if the human does something unexpected. "The robot always has to be aware of the human's presence," Bobick noted.

In many ways, manufacturing is the easiest environment for a robot. An entire factory floor or warehouse can be engineered for automation, with any variables – lighting or floor obstacles – eliminated. Robots working in health care settings or the home don't have such advantages.

Flying farmers

Lora Weiss is taking robotics to new heights – thousands of feet up into the air. Working with bright-yellow, propeller-driven unmanned aerial vehicles (UAVs) that serve as test beds, she's facilitating collaboration between humans and the flying robots – and even among the machines themselves.

The goal is for robots to protect humans from dangerous tasks while leveraging their capabilities. One example is monitoring forests for the early signs of fire. UAVs carrying sensors can quickly cover large areas and easily distinguish a Boy Scout campfire from a growing wildfire. Other examples include search and rescue missions and military tasks that can be accomplished by robots – instead of risking a human pilot.

"Anything that is dangerous, you want the robots to do," said Weiss, who is chief scientist in the Georgia Tech Research Institute (GTRI). "You want the human to be as far away as possible, which means the robots have to go farther on their own."

Like others pursuing co-robotics at Georgia Tech, Weiss sees robots and humans as complementary. "Machines don't get bored like people do," Weiss noted. "Humans know how to put information into context, so we can ask robots to conduct the tedious searches and let people know when they see something unusual."

To be effective, robots will have to be more than mobile sensor systems. Instead of sending a video stream back to an operator who must watch it continuously, an agricultural robot could scan fields, armed with the knowledge of what diseased plants look like. Only when it sees signs of disease would it alert a human collaborator. Having that level of sophistication will require powerful on-board computing able to sense the environment, merge and interpret data, and decide whether the information should be shared.

The robots Weiss works with are fully autonomous, no longer guided by remote pilots. Humans will continue to set mission parameters and oversee the robots, but their involvement will be much less than in the past.

Of course, allowing robots to operate on their own raises ethical questions, particularly in military domains. Just how independent can they be?

"It's like the decisions parents make with teenagers," Weiss said. "You want to give them independence, but only if they're ready for it. When you have confidence that the robot is behaving responsibly, you'll allow it to do more."

Beyond human-robot collaboration, GTRI is also helping teams of UAVs work together. One aircraft may carry a heat sensor, while another will have a video camera. The aircraft work together using their complementary capabilities to provide better information. Recently, GTRI demonstrated that three UAVs could maintain formation while flying together.

"Getting humans and robots to interact the way humans work together is an exciting challenge," said Weiss. "Traditionally, we have had humans commanding the robots and having constant oversight of them. We want to start loosening that control to let the robots do more on their own."

Robotic caregivers

In Charlie Kemp's lab, a robotic arm with sensors in its skin reaches through a table covered with plants. It rustles the leaves as it moves between them, a task unthinkable for robots until recently.

Being able to touch objects – and people who need help – through such cluttered environments will be essential to personal care robots of the future.

"If you decide that the robot's arm shouldn't touch anything, which has been the standard approach, you greatly limit the robot's capabilities," said Kemp. "With these limitations, the robot couldn't reach to the back of a cluttered refrigerator or shelf. Many of the things it could have done for a person with disabilities would be out of reach."

But modern robots can learn and understand much more than earlier generations. Today's robots can know how to avoid harming people, how much force is reasonable to open a door, and how to operate amidst clutter. And research has already shown that people are comfortable with robots touching them.

"With our intelligent control system, we have shown it can be fine for the robot's arm to make contact with people because it keeps the forces low," Kemp explained. "Humans are accustomed to touching the world. We want to give that capability to robots because it can dramatically increase their ability to provide assistance."

Over the past two years, Kemp has been working with Henry and Jane Evans to explore the bounds of personal care robots. When he was just 40 years old, Henry suffered a stroke that left him with the ability to move just his head and a finger. Among their goals for the effort: to make operating a robot simple enough for people without specialized training.

"You shouldn't need a Ph.D. in robotics to be able to program a robot to do these things," said Kemp, who also works with ALS researchers at Emory University in Atlanta. "We want to give robots more intelligence and more autonomy so they'll be easier to use and more useful."

Cost is still an issue, and will be until the price of robotic arms drops further, Kemp said. But through working with more than 200 people, one conclusion is clear:

"People are extremely excited about this technology across the board, even when they have worked with technology that isn't very polished yet," added Kemp, who has funding from the National Science Foundation (NSF) and the National Institute on Disability and Rehabilitation Research (NIDRR). "You can have great technology that could potentially help a lot of people, but if people don't choose to adopt it, it won't do much good."

Conducting a swarm

Magnus Egerstedt works with lots of robots – all at the same time. He's studying how humans can interact with swarms of inexpensive robots that individually have different capabilities.

Swarms offer redundancy and the ability to get big jobs done quickly, but humans aren't very good at managing multiple robots at the same time.

"You have to interact with robots at the team level," explained Egerstedt, the Schlumberger Professor in Georgia Tech's School of Electrical and Computer Engineering. "The questions I'm exploring revolve around how people should be embedded in these swarms and the ways people can direct the robots – much like how a single person conducts an entire orchestra."

The swarms are composed of robots that aren't so capable or talented on their own. But together with other members of the team, they can figure out how to carry out complex tasks – much like swarms of insects. Egerstedt is fond of showing videos in which robot teams form letters of the alphabet, with the team members negotiating among themselves to accomplish the tasks.

"With swarms, you have robots that are autonomous in the sense that they can make their own decisions even if, individually, they are sometimes rather ill-informed," he said. "They have to come together to do things. No single robot is in charge, so they have to negotiate among themselves and with the human operator. The beauty of this is that results emerge."

To make this happen, Egerstedt has to give the robots an ability to think differently than robots have in the past. For instance, a high priority for robots has always been avoiding collisions with other robots or humans. But with dozens, hundreds, or thousands of robots working together, that may no longer be practical.

"When people walk through crowds, we bump into each other all the time," he noted. "We don't run straight into each other, but there are little nudges here and there. Fish and birds also do this. So I'm letting the robots bump into each other. If you allow graceful collisions, you get less conservative solutions and can get more done."

Other challenges will include scaling up the control algorithms from the 15 aerial and ground robots Egerstedt currently has in his lab. He believes that swarms of robots could be useful in search and rescue missions, farming, homeland security, and military operations.

Co-robotics offers benefits in many areas of society. The level of enthusiasm is high, but the potential beneficiaries are tempering their enthusiasm with a dose of reality.

"Like most new technologies, robots will turn out to be extremely useful for some things, and not so useful for others," said Evans. "The point of the current research is to determine what they are best at."

Georgia Tech robotics research is supported by a broad range of organizations, including the National Science Foundation, National Institutes of Health, DARPA, Army Research Office, BMW, Boeing, and John Deere & Company.

John Toon is director of research news at Georgia Tech and editor of Research Horizons magazine. He's been writing about Georgia Tech research and economic development activities for more than 30 years.