Computer scientist seeks the real meaning of language

“The best liars are the people who tell the truth most of the time,” said Hirschberg, who is teaching Computational Approaches to Emotional Speech at the Engineering School this term.

Her latest research project involves working with Barnard psychologist Michelle Levine and Andrew Rosenberg (GSAS’09), a computer science professor at CUNY, to develop computational methods to detect deception in English, Mandarin Chinese and Arabic speakers. Hirschberg hopes that her work can lead to the development of lie detection technology that is more accurate than human intuition or the polygraph, each of which predicts no better than chance.

Her work with deception in speech began with a 2003 study that remains one of the largest collections of such data because, she said, it is so difficult to collect real lies in situations where the truth is known.

For the first half of the study, 32 subjects were asked to complete random, unrelated tasks such as tying knots, stacking quarters on their elbows, hopping on one foot and singing—things they were told America’s top 25 entrepreneurs were also tested on.

“This was a setup for the real experiment,” Hirschberg explained. All the subjects were told they failed to perform like the entrepreneurs but were given the option of demonstrating if they “could talk a good game,” as many successful people do. All opted in and were told to convince an interviewer that they had performed just as well as the entrepreneurs on their tasks, pressing a hidden pedal when they lied and another when they told the truth.

After the recorded interviews, Hirschberg and her colleagues aligned the pedal presses with the recordings, which were then analyzed for a number of speech and language features that could potentially indicate deception, such as pauses, laughter and variation in pitch. Using this data, the researchers built classifiers using machine learning techniques that proved to be about 70 percent accurate in picking out truth from lies on test data. Human judges averaged only 58 percent accuracy when judging the same data.

Hirschberg has used similar approaches to identify charisma conveyed through speech in English, Arabic and Swedish. She is also studying “code switching”—when two bilingual speakers toggle from one language to another mid-conversation.

“How do people convey that it’s another person’s turn to speak? What do people mean when they say ‘okay’? There are so many different ways it’s used,” she said.

Recently, Hirschberg and her students have been combing social media to detect emotions like joy, surprise and sadness. They look for words associated with these emotions in blogs and YouTube video captions, research that could have major applications in advertising, marketing and political campaigns. “When Steve Jobs died, things were sad on the Internet,” she said.

Hirschberg has an unusual background for an engineer. Before she received her Ph.D. in computer science from the University of Pennsylvania, she earned a doctorate in 16th century Mexican social history and taught history at Smith College.

This year she received the International Speech Communication Association’s Medal for Scientific Achievement and the James A. Flanagan Award for Speech and Audio Processing from the Institute of Electrical and Electronics Engineers.

In September, Hirschberg won a nearly $1.5 million grant from the Air Force Office of Scientific Research to study deception in speech across cultures. Her work also has received support from the National Science Foundation, Department of Homeland Security, DARPA and IBM.

“This work is amazingly fun,” said Hirschberg, who worked in the research department of AT&T Laboratories for almost 20 years before joining Columbia in 2002. “It’s an eye-opening experience.”

Provided by Columbia University