Computer scientist seeks the real meaning of language

October 27, 2011 by Meghan Berry, Columbia University
Spoken language expert Julia Hirschberg has received a nearly $1.5 million grant to study deception in speech across cultures. Image credit: Eileen Barroso/Columbia University

Better be careful about telling a lie to Julia Hirschberg. The computer science professor, an expert in spoken language, examines what people unconsciously communicate through such things as intonation, accent and phrasing. One of the enduring questions she studies is whether it’s possible to detect a lie.

“The best liars are the people who tell the truth most of the time,” said Hirschberg, who is teaching Computational Approaches to Emotional Speech at the Engineering School this term.

Her latest research project involves working with Barnard psychologist Michelle Levine and Andrew Rosenberg (GSAS’09), a computer science professor at CUNY, to develop computational methods to detect deception in English, Mandarin Chinese and Arabic speakers. Hirschberg hopes that her work can lead to the development of lie detection technology that is more accurate than human intuition or the polygraph, each of which predicts no better than chance.

Her work with deception in speech began with a 2003 study that remains one of the largest collections of such data because, she said, it is so difficult to collect real lies in situations where the truth is known.

For the first half of the study, 32 subjects were asked to complete random, unrelated tasks such as tying knots, stacking quarters on their elbows, hopping on one foot and singing—things they were told America’s top 25 entrepreneurs were also tested on.

“This was a setup for the real experiment,” Hirschberg explained. All the subjects were told they failed to perform like the entrepreneurs but were given the option of demonstrating if they “could talk a good game,” as many successful people do. All opted in and were told to convince an interviewer that they had performed just as well as the entrepreneurs on their tasks, pressing a hidden pedal when they lied and another when they told the truth.

After the recorded interviews, Hirschberg and her colleagues aligned the pedal presses with the recordings, which were then analyzed for a number of speech and language features that could potentially indicate deception, such as pauses, laughter and variation in pitch. Using this data, the researchers built classifiers using machine learning techniques that proved to be about 70 percent accurate in picking out truth from lies on test data. Human judges averaged only 58 percent accuracy when judging the same data.

Hirschberg has used similar approaches to identify charisma conveyed through speech in English, Arabic and Swedish. She is also studying “code switching”—when two bilingual speakers toggle from one language to another mid-conversation.

“How do people convey that it’s another person’s turn to speak? What do people mean when they say ‘okay’? There are so many different ways it’s used,” she said.

Recently, Hirschberg and her students have been combing social media to detect emotions like joy, surprise and sadness. They look for words associated with these emotions in blogs and YouTube video captions, research that could have major applications in advertising, marketing and political campaigns. “When Steve Jobs died, things were sad on the Internet,” she said.

Hirschberg has an unusual background for an engineer. Before she received her Ph.D. in from the University of Pennsylvania, she earned a doctorate in 16th century Mexican social history and taught history at Smith College.

This year she received the International Speech Communication Association’s Medal for Scientific Achievement and the James A. Flanagan Award for Speech and Audio Processing from the Institute of Electrical and Electronics Engineers.

In September, Hirschberg won a nearly $1.5 million grant from the Air Force Office of Scientific Research to study deception in speech across cultures. Her work also has received support from the National Science Foundation, Department of Homeland Security, DARPA and IBM.

“This work is amazingly fun,” said Hirschberg, who worked in the research department of AT&T Laboratories for almost 20 years before joining Columbia in 2002. “It’s an eye-opening experience.”

Explore further: 'Motherese' important for children's language development

Related Stories

'Motherese' important for children's language development

May 6, 2011

(Medical Xpress) -- Talking to children has always been fundamental to language development, but new research reveals that the way we talk to children is key to building their ability to understand and create sentences of ...

Professor says hate speech is akin to obscenity

January 25, 2011

Obscenity isn’t about sex, but rather about degradation, argues a Michigan State University professor who conducted a first-of-its kind study on the relationship between hate speech and obscenity.

Is the Internet lying to us?

November 25, 2010

(PhysOrg.com) -- University of Alberta scholars talk about the relativity of truth on the World Wide Web.

'The King's Speech': good drama - but accurate science?

February 21, 2011

(PhysOrg.com) -- "The King's Speech" is a compelling enough story to merit 12 Oscar nominations. (We’ll find out how compelling when the Academy Awards are announced Feb. 27). However, as contentions surface about the ...

The kids are alright

May 26, 2011

Children should be seen and not heard... who says? A Philosophy academic at The University of Nottingham is challenging the adage by teaching primary school children to argue properly.

Recommended for you

Permanent, wireless self-charging system using NIR band

October 8, 2018

As wearable devices are emerging, there are numerous studies on wireless charging systems. Here, a KAIST research team has developed a permanent, wireless self-charging platform for low-power wearable electronics by converting ...

Facebook launches AI video-calling device 'Portal'

October 8, 2018

Facebook on Monday launched a range of AI-powered video-calling devices, a strategic revolution for the social network giant which is aiming for a slice of the smart speaker market that is currently dominated by Amazon and ...

Artificial enzymes convert solar energy into hydrogen gas

October 4, 2018

In a new scientific article, researchers at Uppsala University describe how, using a completely new method, they have synthesised an artificial enzyme that functions in the metabolism of living cells. These enzymes can utilize ...

6 comments

Adjust slider to filter visible comments by rank

Display comments: newest first

Grizzled
not rated yet Oct 28, 2011
Well, when Air Force grants you 1.5mil buckeroos to find out the best way to catch a cheat, I guess they mean business :-)
Callippo
not rated yet Oct 28, 2011
The real meaning of language isn't definitely to detect a lie, but to express and cover it. I'm affraid the research of Mrs. Hirschberg suffers with many hidden paradoxes.
Isaacsname
5 / 5 (1) Oct 28, 2011
Mairzy doats and dozy doats and liddle lamzy divey
A kiddley divey too, wooden shoe ?

Fleas two meat chew, I eggs cell at bean the perfect toast. Oui C'lang wedge is spore attic buttle ways cohere rent.

..wait...maybe
alfie_null
not rated yet Oct 29, 2011
I'm looking forward to when I can put a veracity app on my phone. Of course by then everyone will be employing virtual avatars to filter what they say.
Isaacsname
not rated yet Oct 30, 2011
For those interested, mondegreens, soramimi, and homophonic translations. I wonder, Hush, is there a proper term for the phenomenon of a linguistic " illusion " ? I could compose a paragraph of seemingly nonsensical phrases that sound like coherent speech to a listener, but appear to be gibberish to the reader.
larryp
not rated yet Oct 31, 2011
Hearing is not sound. It's the brains interpretation of sound. Instances of the brain mis-interpreting sounds are common. It is pretty well established that "what you hear" can be modified by what you see, smell or feel (e.g. how much fear is present). I guess you could call it an earllusion.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.