Computer scientist seeks the real meaning of language

Oct 27, 2011 by Meghan Berry
Spoken language expert Julia Hirschberg has received a nearly $1.5 million grant to study deception in speech across cultures. Image credit: Eileen Barroso/Columbia University

Better be careful about telling a lie to Julia Hirschberg. The computer science professor, an expert in spoken language, examines what people unconsciously communicate through such things as intonation, accent and phrasing. One of the enduring questions she studies is whether it’s possible to detect a lie.

“The best liars are the people who tell the truth most of the time,” said Hirschberg, who is teaching Computational Approaches to Emotional Speech at the Engineering School this term.

Her latest research project involves working with Barnard psychologist Michelle Levine and Andrew Rosenberg (GSAS’09), a computer science professor at CUNY, to develop computational methods to detect deception in English, Mandarin Chinese and Arabic speakers. Hirschberg hopes that her work can lead to the development of lie detection technology that is more accurate than human intuition or the polygraph, each of which predicts no better than chance.

Her work with deception in speech began with a 2003 study that remains one of the largest collections of such data because, she said, it is so difficult to collect real lies in situations where the truth is known.

For the first half of the study, 32 subjects were asked to complete random, unrelated tasks such as tying knots, stacking quarters on their elbows, hopping on one foot and singing—things they were told America’s top 25 entrepreneurs were also tested on.

“This was a setup for the real experiment,” Hirschberg explained. All the subjects were told they failed to perform like the entrepreneurs but were given the option of demonstrating if they “could talk a good game,” as many successful people do. All opted in and were told to convince an interviewer that they had performed just as well as the entrepreneurs on their tasks, pressing a hidden pedal when they lied and another when they told the truth.

After the recorded interviews, Hirschberg and her colleagues aligned the pedal presses with the recordings, which were then analyzed for a number of speech and language features that could potentially indicate deception, such as pauses, laughter and variation in pitch. Using this data, the researchers built classifiers using machine learning techniques that proved to be about 70 percent accurate in picking out truth from lies on test data. Human judges averaged only 58 percent accuracy when judging the same data.

Hirschberg has used similar approaches to identify charisma conveyed through speech in English, Arabic and Swedish. She is also studying “code switching”—when two bilingual speakers toggle from one language to another mid-conversation.

“How do people convey that it’s another person’s turn to speak? What do people mean when they say ‘okay’? There are so many different ways it’s used,” she said.

Recently, Hirschberg and her students have been combing social media to detect emotions like joy, surprise and sadness. They look for words associated with these emotions in blogs and YouTube video captions, research that could have major applications in advertising, marketing and political campaigns. “When Steve Jobs died, things were sad on the Internet,” she said.

Hirschberg has an unusual background for an engineer. Before she received her Ph.D. in from the University of Pennsylvania, she earned a doctorate in 16th century Mexican social history and taught history at Smith College.

This year she received the International Speech Communication Association’s Medal for Scientific Achievement and the James A. Flanagan Award for Speech and Audio Processing from the Institute of Electrical and Electronics Engineers.

In September, Hirschberg won a nearly $1.5 million grant from the Air Force Office of Scientific Research to study deception in speech across cultures. Her work also has received support from the National Science Foundation, Department of Homeland Security, DARPA and IBM.

“This work is amazingly fun,” said Hirschberg, who worked in the research department of AT&T Laboratories for almost 20 years before joining Columbia in 2002. “It’s an eye-opening experience.”

Explore further: New algorithm identifies data subsets that will yield the most reliable predictions

add to favorites email to friend print save as pdf

Related Stories

'Motherese' important for children's language development

May 06, 2011

(Medical Xpress) -- Talking to children has always been fundamental to language development, but new research reveals that the way we talk to children is key to building their ability to understand and create ...

Professor says hate speech is akin to obscenity

Jan 25, 2011

Obscenity isn’t about sex, but rather about degradation, argues a Michigan State University professor who conducted a first-of-its kind study on the relationship between hate speech and obscenity.

Is the Internet lying to us?

Nov 25, 2010

(PhysOrg.com) -- University of Alberta scholars talk about the relativity of truth on the World Wide Web.

'The King's Speech': good drama - but accurate science?

Feb 21, 2011

(PhysOrg.com) -- "The King's Speech" is a compelling enough story to merit 12 Oscar nominations. (We’ll find out how compelling when the Academy Awards are announced Feb. 27). However, as contentions surface about the ...

The kids are alright

May 26, 2011

Children should be seen and not heard... who says? A Philosophy academic at The University of Nottingham is challenging the adage by teaching primary school children to argue properly.

Recommended for you

Designing exascale computers

Jul 23, 2014

"Imagine a heart surgeon operating to repair a blocked coronary artery. Someday soon, the surgeon might run a detailed computer simulation of blood flowing through the patient's arteries, showing how millions ...

User comments : 14

Adjust slider to filter visible comments by rank

Display comments: newest first

hush1
not rated yet Oct 27, 2011
lol
Meghan Berry? The jig is up. For the sake of your readers let them (eventually) know how much truth is in the article.
I already know.
Grizzled
not rated yet Oct 28, 2011
Well, when Air Force grants you 1.5mil buckeroos to find out the best way to catch a cheat, I guess they mean business :-)
hush1
not rated yet Oct 28, 2011
Thank you. You fell for it. :)
Callippo
not rated yet Oct 28, 2011
The real meaning of language isn't definitely to detect a lie, but to express and cover it. I'm affraid the research of Mrs. Hirschberg suffers with many hidden paradoxes.
Isaacsname
5 / 5 (1) Oct 28, 2011
Mairzy doats and dozy doats and liddle lamzy divey
A kiddley divey too, wooden shoe ?

Fleas two meat chew, I eggs cell at bean the perfect toast. Oui C'lang wedge is spore attic buttle ways cohere rent.

..wait...maybe
hush1
not rated yet Oct 28, 2011
Not quite, Isaacsname. Your attempt at incoherence is more than you care to reveal!

Callippo. You can not give words 'cover'. The greatest 'lie detectors' are words used that (in any language) have the least meaning for the speaker and the listener!

Additionally, these 'researchers' relied on acoustical and visual cues!!


electrodynamic
not rated yet Oct 29, 2011
Sounds like she is trying to build a database of some sort.
The computer doesn't understand the conversation taking place on the level a human does, or all the hidden agendas a person may have. The computer can only try to match new data to similar data, predict the probability this is the same situation, and assign a value to some statement. If
there is a perfect lie detector out there, it will be covered up by the people that have the most to fear from the truth. Lawyers, and Politicians.
alfie_null
not rated yet Oct 29, 2011
I'm looking forward to when I can put a veracity app on my phone. Of course by then everyone will be employing virtual avatars to filter what they say.
hush1
not rated yet Oct 29, 2011
If there is a perfect lie detector out there, it will be covered up by the people that have the most to fear from the truth. Lawyers, and Politicians. - electrodynamic

Not quite. Close. The rhetoric of those groups contain the perfect lie detectors words and wording. The excessive usage of those words and wordings are typical of professions that must manipulate truth values to obtain a desired outcome. The profession most likely to recognize this is a branch of study of language called:
Psycholinguistics.

To date, those that study psycholinguistics have not come upon the idea of looking into the impact of the meanings of words in the context of truth values.

I hope they never do. It is bad enough that lawyers, politicians, coverts, etc., used this without being actively cognitively aware of the manipulation. If this were to become a branch of science and study, the risk of abuse can not be constrained.

The researchers are without my support. No price tag will sway me.
Isaacsname
not rated yet Oct 30, 2011
For those interested, mondegreens, soramimi, and homophonic translations. I wonder, Hush, is there a proper term for the phenomenon of a linguistic " illusion " ? I could compose a paragraph of seemingly nonsensical phrases that sound like coherent speech to a listener, but appear to be gibberish to the reader.
hush1
not rated yet Oct 30, 2011
Psychologists will label that a variation of neologism.
I will label that an association (also in the psychological usage and sense). Such associations are common place. The purpose is to assist imaginative associations.

I am looking into your question if what you describe has been an official label.
hush1
not rated yet Oct 30, 2011
I will conjecture, for humans, it is not possible to produce or hear a sound without meaning.

I will further conjecture that sound is fundamental to Nature.

If you assert the opposite of both conjectures above, then a purpose for sound can never be found.
larryp
not rated yet Oct 31, 2011
Hearing is not sound. It's the brains interpretation of sound. Instances of the brain mis-interpreting sounds are common. It is pretty well established that "what you hear" can be modified by what you see, smell or feel (e.g. how much fear is present). I guess you could call it an earllusion.
hush1
not rated yet Nov 16, 2011
Hearing is not sound. - larryp

Correct.
There are those that experience sound(hear)when seeing motion.
There are those that experience sound(hear)upon being touched.

I am not ready to call the percepts send by the senses to the brain illusion.