Follow the eyes: Head-mounted cameras could help robots understand social interactions

Dec 13, 2012

(Phys.org)—What is everyone looking at? It's a common question in social settings because the answer identifies something of interest, or helps delineate social groupings. Those insights someday will be essential for robots designed to interact with humans, so researchers at Carnegie Mellon University's Robotics Institute have developed a method for detecting where people's gazes intersect.

The researchers tested the method using groups of people with head-mounted . By noting where their gazes converged in three-dimensional space, the researchers could determine if they were listening to a single speaker, interacting as a group, or even following the in a ping-pong game.

The system thus uses crowdsourcing to provide subjective information about that would otherwise be difficult or impossible for a robot to ascertain.

The researchers' algorithm for determining "social saliency" could ultimately be used to evaluate a variety of , such as the expressions on people's faces or , or data from other types of visual or audio sensors.

"This really is just a first step toward analyzing the of people," said Hyun Soo Park, a Ph.D. student in mechanical engineering, who worked on the project with Yaser Sheikh, assistant research professor of robotics, and Eakta Jain of Texas Instruments, who was awarded a Ph.D. in robotics last spring. "In the future, robots will need to interact organically with people and to do so they must understand their , not just their ."

Though head-mounted cameras are still unusual, police officers, soldiers, search-and-rescue personnel and even surgeons are among those who have begun to wear body-mounted cameras. Head-mounted systems, such as those integrated into eyeglass frames, are poised to become more common. Even if person-mounted cameras don't become ubiquitous, Sheikh noted that these cameras someday might be used routinely by people who work in cooperative teams with robots.

The technique was tested in three real-world settings: a meeting involving two work groups; a musical performance; and a party in which participants played pool and ping-pong and chatted in small groups.

The head-mounted cameras provided precise data about what people were looking at in . The algorithm developed by the research team was able to automatically estimate the number and 3D position of "gaze concurrences"—positions where the gazes of multiple people intersected.

But the researchers were surprised by the level of detail they were able to detect. In the party setting, for instance, the algorithm didn't just indicate that people were looking at the ping-pong table; the gaze concurrence video actually shows the flight of the ball as it bounces and is batted back and forth.

That finding suggests another possible application for monitoring gaze concurrence: player-level views of ball games. Park said if basketball players all wore head-mounted cameras, for instance, it might be possible to reconstruct the game, not from the point of view of a single player, but from a collective view of the players as they all keep their eyes on the ball.

Another potential use is the study of social behavior, such as group dynamics and gender interactions, and research into behavioral disorders, such as autism.

Explore further: UT Dallas professor to develop framework to protect computers' cores

More information: More information on gaze concurrence, including a video, is available on the project website. The researchers reported their findings Dec. 3 at the Neural Information Processing Systems Conference in Lake Tahoe, Nev

Related Stories

Ping-pong robots debut in China (w/ video)

Oct 15, 2011

(PhysOrg.com) -- Last week some oohs and ahhs were in order as two ping-pong playing robots made their debut at Zhejiang University in China. The two robots played against each other and with humans. True, ...

Body-mounted cameras turn motion capture inside out

Aug 08, 2011

Traditional motion capture techniques use cameras to meticulously record the movements of actors inside studios, enabling those movements to be translated into digital models. But by turning the cameras around — mounting ...

Recommended for you

User comments : 0

More news stories

Ex-Apple chief plans mobile phone for India

Former Apple chief executive John Sculley, whose marketing skills helped bring the personal computer to desktops worldwide, says he plans to launch a mobile phone in India to exploit its still largely untapped ...

Airbnb rental site raises $450 mn

Online lodging listings website Airbnb inked a $450 million funding deal with investors led by TPG, a source close to the matter said Friday.

Health care site flagged in Heartbleed review

People with accounts on the enrollment website for President Barack Obama's signature health care law are being told to change their passwords following an administration-wide review of the government's vulnerability to the ...

A homemade solar lamp for developing countries

(Phys.org) —The solar lamp developed by the start-up LEDsafari is a more effective, safer, and less expensive form of illumination than the traditional oil lamp currently used by more than one billion people ...

NASA's space station Robonaut finally getting legs

Robonaut, the first out-of-this-world humanoid, is finally getting its space legs. For three years, Robonaut has had to manage from the waist up. This new pair of legs means the experimental robot—now stuck ...

Filipino tests negative for Middle East virus

A Filipino nurse who tested positive for the Middle East virus has been found free of infection in a subsequent examination after he returned home, Philippine health officials said Saturday.

Egypt archaeologists find ancient writer's tomb

Egypt's minister of antiquities says a team of Spanish archaeologists has discovered two tombs in the southern part of the country, one of them belonging to a writer and containing a trove of artifacts including reed pens ...