Read my lips: New technology spells out what's said when audio fails

March 24, 2016
smile

New lip-reading technology developed at the University of East Anglia (UEA) could help in solving crimes and provide communication assistance for people with hearing and speech impairments.

The visual , created by Dr Helen L. Bear and Prof Richard Harvey of UEA's School of Computing Sciences, can be applied "any place where the audio isn't good enough to determine what people are saying," Dr Bear said.

Dr Bear, whose findings will be presented at the International Conference on Acoustics, Speech and Signal Processing (ICASSP) in Shanghai on March 25, said unique problems with determining arise when sound isn't available - such as on CCTV footage - or if the audio is inadequate and there aren't clues to give the context of a conversation. The sounds '/p/,' '/b/,' and '/m/' all look similar on the lips, but now the machine lip-reading classification technology can differentiate between the sounds for a more accurate translation.

Dr Bear said: "We are still learning the science of visual speech and what it is people need to know to create a fool-proof recognition model for lip-reading, but this classification system improves upon previous lip-reading methods by using a novel training method for the classifiers.

"Potentially, a robust lip-reading system could be applied in a number of situations, from criminal investigations to entertainment. Lip-reading has been used to pinpoint words footballers have shouted in heated moments on the pitch, but is likely to be of most practical use in situations where are there are high levels of noise, such as in cars or aircraft cockpits.

"Crucially, whilst there are still improvements to be made, such a system could be adapted for use for a range of purposes - for example, for people with hearing or speech impairments. Alternatively, a good lip-reading machine could be part of an audio-visual recognition system."

Prof Harvey said: "Lip-reading is one of the most challenging problems in artificial intelligence so it's great to make progress on one of the trickier aspects, which is how to train machines to recognise the appearance and shape of human lips."

The research was part of a three-year project and was supported by the Engineering and Physical Sciences Research Council (EPSRC).

The paper, Decoding visemes: Improving machine lip-reading, will be published on March 25, 2016 in the Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing 2016.

Explore further: Lip-read me now, hear me better later

Related Stories

Lip-read me now, hear me better later

April 12, 2007

Experience hearing a person's voice allows us to more easily hear what they are saying. Now research by UC Riverside psychology Professor Lawrence D. Rosenblum and graduate students Rachel M. Miller and Kauyumari Sanchez ...

Recommended for you

Sponge creates steam using ambient sunlight

August 22, 2016

How do you boil water? Eschewing the traditional kettle and flame, MIT engineers have invented a bubble-wrapped, sponge-like device that soaks up natural sunlight and heats water to boiling temperatures, generating steam ...

4 comments

Adjust slider to filter visible comments by rank

Display comments: newest first

IronhorseA
5 / 5 (1) Mar 24, 2016
Strange, the picture doesn't look like George H. Bush. D~
pntaylor
5 / 5 (2) Mar 25, 2016
Strange, the picture doesn't look like George H. Bush. D~

That's because George talked out his ass.
IronhorseA
not rated yet Mar 25, 2016
Strange, the picture doesn't look like George H. Bush. D~

That's because George talked out his ass.

What a strange place to use lipstick. D~
betterexists
not rated yet Apr 18, 2016
Good!

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.