Read my lips: New technology spells out what's said when audio fails

March 24, 2016

New lip-reading technology developed at the University of East Anglia (UEA) could help in solving crimes and provide communication assistance for people with hearing and speech impairments.

The visual , created by Dr Helen L. Bear and Prof Richard Harvey of UEA's School of Computing Sciences, can be applied "any place where the audio isn't good enough to determine what people are saying," Dr Bear said.

Dr Bear, whose findings will be presented at the International Conference on Acoustics, Speech and Signal Processing (ICASSP) in Shanghai on March 25, said unique problems with determining arise when sound isn't available - such as on CCTV footage - or if the audio is inadequate and there aren't clues to give the context of a conversation. The sounds '/p/,' '/b/,' and '/m/' all look similar on the lips, but now the machine lip-reading classification technology can differentiate between the sounds for a more accurate translation.

Dr Bear said: "We are still learning the science of visual speech and what it is people need to know to create a fool-proof recognition model for lip-reading, but this classification system improves upon previous lip-reading methods by using a novel training method for the classifiers.

"Potentially, a robust lip-reading system could be applied in a number of situations, from criminal investigations to entertainment. Lip-reading has been used to pinpoint words footballers have shouted in heated moments on the pitch, but is likely to be of most practical use in situations where are there are high levels of noise, such as in cars or aircraft cockpits.

"Crucially, whilst there are still improvements to be made, such a system could be adapted for use for a range of purposes - for example, for people with hearing or speech impairments. Alternatively, a good lip-reading machine could be part of an audio-visual recognition system."

Prof Harvey said: "Lip-reading is one of the most challenging problems in artificial intelligence so it's great to make progress on one of the trickier aspects, which is how to train machines to recognise the appearance and shape of human lips."

The research was part of a three-year project and was supported by the Engineering and Physical Sciences Research Council (EPSRC).

The paper, Decoding visemes: Improving machine lip-reading, will be published on March 25, 2016 in the Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing 2016.

Explore further: Findings could lead to improved lip-reading training for the deaf and hard-of-hearing

Related Stories

Lip-read me now, hear me better later

April 12, 2007

Experience hearing a person's voice allows us to more easily hear what they are saying. Now research by UC Riverside psychology Professor Lawrence D. Rosenblum and graduate students Rachel M. Miller and Kauyumari Sanchez ...

Recommended for you

Making it easier to collaborate on code

October 26, 2016

Git is an open-source system with a polarizing reputation among programmers. It's a powerful tool to help developers track changes to code, but many view it as prohibitively difficult to use.

Dutch unveil giant vacuum to clean outside air

October 25, 2016

Dutch inventors Tuesday unveiled what they called the world's first giant outside air vacuum cleaner—a large purifying system intended to filter out toxic tiny particles from the atmosphere surrounding the machine.


Adjust slider to filter visible comments by rank

Display comments: newest first

5 / 5 (1) Mar 24, 2016
Strange, the picture doesn't look like George H. Bush. D~
5 / 5 (2) Mar 25, 2016
Strange, the picture doesn't look like George H. Bush. D~

That's because George talked out his ass.
not rated yet Mar 25, 2016
Strange, the picture doesn't look like George H. Bush. D~

That's because George talked out his ass.

What a strange place to use lipstick. D~
not rated yet Apr 18, 2016

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.