Sign language over a mobile phone

Aug 22, 2008

A group at the University of Washington has developed software that for the first time enables deaf and hard-of-hearing Americans to use sign language over a mobile phone. UW engineers got the phones working together this spring, and recently received a National Science Foundation grant for a 20-person field project that will begin next year in Seattle.

This is the first time two-way real-time video communication has been demonstrated over cell phones in the United States. Since posting a video of the working prototype on YouTube, deaf people around the country have been writing on a daily basis.

"A lot of people are excited about this," said principal investigator Eve Riskin, a UW professor of electrical engineering. For mobile communication, deaf people now communicate by cell phone using text messages. "But the point is you want to be able to communicate in your native language," Riskin said. "For deaf people that's American Sign Language."

Video is much better than text-messaging because it's faster and it's better at conveying emotion, said Jessica DeWitt, a UW undergraduate in psychology who is deaf and is a collaborator on the MobileASL project. She says a large part of her communication is with facial expressions, which are transmitted over the video phones.

Low data transmission rates on U.S. cellular networks, combined with limited processing power on mobile devices, have so far prevented real-time video transmission with enough frames per second that it could be used to transmit sign language. Communication rates on United States cellular networks allow about one tenth of the data rates common in places such as Europe and Asia (sign language over cell phones is already possible in Sweden and Japan).

Even as faster networks are becoming more common in the United States, there is still a need for phones that would operate on the slower systems.

"The faster networks are not available everywhere," said doctoral student Anna Cavender. "They also cost more. We don't think it's fair for someone who's deaf to have to pay more for his or her cell phone than someone who's hearing."

The team tried different ways to get comprehensible sign language on low-resolution video. They discovered that the most important part of the image to transmit in high resolution is around the face. This is not surprising, since eye-tracking studies have already shown that people spend the most time looking at a person's face while they are signing.

The current version of MobileASL uses a standard video compression tool to stay within the data transmission limit. Future versions will incorporate custom tools to get better quality. The team developed a scheme to transmit the person's face and hands in high resolution, and the background in lower resolution. Now they are working on another feature that identifies when people are moving their hands, to reduce battery consumption and processing power when the person is not signing.

The team is currently using phones imported from Europe, which are the only ones they could find that would be compatible with the software and have a camera and video screen located on the same side of the phone so that people can film themselves while watching the screen.

Mobile video sign language won't be widely available until the service is provided through a commercial cell-phone manufacturer, Riskin said. The team has already been in discussion with a major cellular network provider that has expressed interest in the project.

Source: University of Washington

Explore further: Mandela phone app for tourists in S.Africa

Related Stories

New app first to use gesture for language learning

Jan 29, 2015

While you might think a person shaking her phone or tablet from side to side is having issues with the device, she might actually be playing a game that has her mimicking a steering wheel motion as part of ...

Now Skype can translate for us, why learn a language?

Jan 14, 2015

New technology has the habit of making certain professions redundant. Power looms put cotton workers out of job, leading to the rise of the Luddites. Word processors put an end to the typing pool. Now fre ...

Recommended for you

Mandela phone app for tourists in S.Africa

1 hour ago

A phone app tracing the footsteps of Nelson Mandela was launched Wednesday in South Africa to encourage tourists to explore his life story, 25 years after his release from prison.

App to test synaesthesia

2 hours ago

Four in hundred people have a special mix up of their senses, called synaesthesia. A new app from Radboud University contains four playful tests for synaesthesia. Should you happen to have synaesthesia, you ...

Windows Insiders can try out Project Spartan browser

Mar 31, 2015

Microsoft has opened up the (literal) windows, called in creatives, and has been engineering a next-generation browser. Project Spartan is to reflect the general mood of fresh air at Redmond. Although "Project ...

New taxi app challenges Uber in S.Korea

Mar 31, 2015

South Korea's top mobile messenger operator launched a new web-based cab-hailing service Tuesday to compete with California-based Uber, whose service has been subjected to crackdowns from state regulators.

User comments : 0

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.