New software for Google Glass provides captions for hard-of-hearing users

October 2, 2014, Georgia Institute of Technology
Captioning on Glass allows a person to speak into a phone. The text is then displayed on Glass. Credit: Georgia Tech

A team of Georgia Institute of Technology researchers has created speech-to-text software for Google Glass that helps hard-of-hearing users with everyday conversations. A hard-of-hearing person wears Glass while a second person speaks directly into a smartphone. The speech is converted to text, sent to Glass and displayed on its heads-up display.

A group in Georgia Tech's College of Computing created the Glassware when one of its own said he was having trouble hearing and thought Glass could help him.

"This system allows wearers like me to focus on the speaker's lips and facial gestures, "said School of Interactive Computing Professor Jim Foley. "If hard-of-hearing people understand the speech, the conversation can continue immediately without waiting for the caption. However, if I miss a word, I can glance at the transcription, get the word or two I need and get back into the conversation."

Foley's colleague, Professor Thad Starner, leads the Contextual Computing Group working on the project. He says using a with Glass has several benefits as compared to using Glass by itself.

"Glass has its own microphone, but it's designed for the wearer," said Starner, who is also a technical lead for Glass. "The mobile phone puts a microphone directly next to the speaker's mouth, reducing background noise and helping to eliminate errors."

Starner says the phone-to-Glass system is helpful because speakers are more likely to construct their sentences more clearly, avoiding "uhs" and "ums." However, if captioning errors are sent to Glass, the smartphone software also allows the speaker to edit the mistakes, sending the changes to the person wearing the device.

"The smartphone uses the Android transcription API to convert the audio to text," said Jay Zuerndorfer, the Georgia Tech Computer Science graduate student who developed the software. "The text is then streamed to Glass in real time."

Captioning on Glass is currently available to install from MyGlass. More information and support can be found at the project website here.

A hard-of-hearing person wears Glass while a second person speaks directly into a smartphone. The speech is converted to text, sent to Glass and displayed on its heads-up display. Credit: Georgia Tech

Foley and the students are working with the Association of Late Deafened Adults in Atlanta to improve the program.

The same group is also working on a second project, Translation on Glass, that uses the same smartphone-Glass Bluetooth connection process to capture sentences spoken into the smartphone, translate them to another language and send them to Glass. The only difference is that the person wearing Glass, after reading the translation, can reply. The response is translated back to the original language on the smartphone. Two-way translations are currently available for English, Spanish, French, Russian, Korean and Japanese.

Georgia Tech Professor Jim Foley has trouble hearing and wanted an app that could help him. He and his team developed Captioning on Glass, which is now available on MyGlass. Credit: Georgia Tech
"For both uses, the person wearing Glass has to hand their smartphone to someone else to begin a conversation," said Starner. "It's not ideal for strangers, but we designed the program to be used among friends, trusted acquaintances or while making purchases."

The group is working to get Translation on Glass ready for the public.

The app just became available for the public on October 2.

Explore further: Drivers, don't trade in your smartphone for Google Glass yet

Related Stories

Drivers, don't trade in your smartphone for Google Glass yet

September 24, 2014

Texting while driving with Google Glass is clearly a distraction, a new University of Central Florida study has concluded—but there is a twist. In the study, texting Glass users outperformed smartphone users when regaining ...

SHORE facial analysis spots emotions on Google Glass

August 28, 2014

One of the key concerns about facial recognition software has been over privacy. The very idea of having tracking mechanisms as part of an Internet-connected wearable would be likely to upset many privacy advocates. German ...

Recommended for you

After a reset, Сuriosity is operating normally

February 23, 2019

NASA's Curiosity rover is busy making new discoveries on Mars. The rover has been climbing Mount Sharp since 2014 and recently reached a clay region that may offer new clues about the ancient Martian environment's potential ...

Study: With Twitter, race of the messenger matters

February 23, 2019

When NFL player Colin Kaepernick took a knee during the national anthem to protest police brutality and racial injustice, the ensuing debate took traditional and social media by storm. University of Kansas researchers have ...


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.