New translator app makes sense of foreign-language food menus

"You type in the menu listing and the application translates it automatically without talking to a server," said Mireille "Mimi" Boutin, an associate professor in Purdue University's School of Electrical and Computer Engineering. "It only takes a fraction of a second, you don't need connection to the Internet and it won't empty your battery."

Before entering a foreign country, the user would download a region- and language-specific configuration and database. From then on, the system can operate without a network connection.

"The problem with menus is that even if you know the language you may still have to ask questions to clarify what a dish contains," Boutin said. "For example, in German, "Schinken" means ham, but it can be raw ham or cooked ham. If you are going to eat the ham, you might want to know which."

The user types the desired dish into a prompt field in the graphical user interface. The text is translated, and the best possible translations are then listed, along with other information, including pictures and ingredients. The user can then browse the multimedia database to obtain more information about the dish or the ingredients. When appropriate, information and questions for the waiter are suggested.

"With Albert Parra Pozo, a graduate student who is fluent in Spanish, we were able to develop and implement this system on the iPod Touch for English speakers traveling in Spain," said Boutin, who specializes in signal and image processing. "Our tests indicate that our system yields a correct translation more often than general-purpose translation engines. Moreover, it does so almost instantaneously. The memory requirements of the application, including the database of pictures, are also well within the limits of the device."

The researchers used a method called n-gram consolidation that improves the translation accuracy and dramatically reduces the size of the database while increasing search speed.

"As a result, our application can be used in a real-time, network-independent environment and produce highly accurate results," Boutin said. "We have a prototype in Spanish and have demonstrated the translation can be done in real time within a fraction of a second.

Findings were detailed in a paper presented in July at the IEEE International Conference on Multimedia and Expo in Barcelona, Spain. The paper was written by Parra, graduate student Andrew W. Haddad, Boutin and Edward J. Delp, the Charles William Harrison Distinguished Professor of Electrical and Computer Engineering.

"We believe the approach we are taking in this project is extendable to other types of mobile devices, including iPads and also the Android mobile phones," Delp said. "The results have been extremely good."

The researchers developed "lightweight algorithms" that operate quickly, have low-energy consumption and require low memory. The real-time translation is nine-hundreds of a second on average, and the application has a memory size of 9.56 megabytes, including its multimedia database, compared to several gigabytes for conventional translation systems, Boutin said.

The method could be further developed as a tool for people who have special diets by adding a database of nutritional information.

"People who must follow a medical diet are often reluctant to travel for fear of putting their health at risk," Boutin said. "Without the ability to understand menus, it is impossible to make informed food choices. In some extreme cases, for example peanut allergies, the consumption of even a minute quantity of certain nutrients can be fatal. In other cases, such as diabetes or congenital metabolic conditions like PKU, the consumption of certain nutrients must be carefully monitored and limited in order to maintain an individual's health."

The system warns people when certain ingredients or nutrients are contained in a dish. When there is doubt about an ingredient's safety, a warning symbol is displayed next to the dish and the device suggests questions and answers, written in both languages, so the user and the restaurant staff can discuss the content of the dish and possible alternatives.

The translator displays a sentence for the patron to read to the server.

"It could be something like, 'Excuse me sir, I have an allergy and was wondering if it's possible to remove the croutons from the salad,'" Boutin said.

The researchers are sometimes able to modify existing translation algorithms.

"It would be easier if we could simply use existing language translation tools," Boutin said. "But these have mostly been developed for desktops, which are directly connected to an outlet and have a large amount of memory and CPU available. In order to make the app work on a cellular phone, we need to rethink everything and develop our own custom-designed methods."

More information: A Hand-held Multimedia Translation and Interpretation System for Diet Management

ABSTRACT
We propose a system for helping individuals who follow a medical diet maintain this diet while visiting countries where a foreign language is spoken. Our focus is on diets where certain foods must either be restricted (e.g., metabolic diseases), avoided (e.g., food intolerance or allergies), or preferably consumed for medical reasons. However, our framework can be used to manage other diets (e.g., vegan) as well. The system is based on the use of a hand-held multimedia device such as a PDA or mobile telephone to analyze and/or disambiguate the content of foods offered on restaurant menus and interpret them in the context of specific diets. The system also provides the option to communicate diet-related instructions or information to a local person (e.g., a waiter) as well as obtain clarifications through dialogue. All computations are performed within the device and do not require a network connection. Real-time text translation is a challenge. We address this challenge with a light-weight, context-specific machine translation method. This method builds on a modification of existing open source Machine Translation (MT) software to obtain a fast and accurate translation. In particular, we describe a method we call n-gram consolidation that joins words in a language pair and increases the accuracy of the translation. We developed and implemented this system on the iPod Touch for English speakers traveling in Spain. Our tests indicate that our translation method yields the correct translation more often than general-purpose translation engines such as Google Translate, and does so almost instantaneously. The memory requirements of the application, including the database of picture, are also well within the limits of the device.

Provided by Purdue University