Mobile control with facial gestures

July 4, 2017, Fraunhofer-Gesellschaft
Test person with EarFS prototype controlling the mobile phone using facial gestures. Credit: © Photo Fraunhofer IGD

Mobile devices play an increasingly important role in our lives; in some situations, though, they cannot be adequately managed and even accepting a call is a real challenge. In a study, researchers at the Fraunhofer Institute for Computer Graphics Research IGD in Rostock, Germany have evaluated which alternative control concepts are suitable to supplement the conventional use of mobile devices. EarFieldSensing (EarFS), the proprietary development that recognizes facial gestures via a special ear plug, has a lot of potential and offers further development possibilities besides its use on mobile devices.

Modern are usually controlled by means of a touch screen. In , however, there are many situations in which this type of operation cannot be implemented. If you are wearing gloves or have your hands full with purchases, the use of smartphones, etc. is difficult, at best. Scientists from the Fraunhofer IGD are therefore looking for alternative concepts for managing mobile devices. Operation via language is an obvious choice. However, challenges such as environmental noise as well as social acceptance impose narrow limits on language-based operation. The Fraunhofer experts' solution: operation via head and face gestures, such as winking, smiling or nodding.

EarFS measures a smile in a person's ear

In the context of researching contactless operation for mobile scenarios, the Rostock researchers have evaluated various technologies with which the movements of a person's head and face can be interpreted. In this process, great importance was placed on practical use in everyday life. For example, systems that read gestures using sensors directly on one's face are very accurate and able to recognize a large number of gestures. They are so conspicuous and unpleasant to wear, though, that they are not suitable for daily use in public. Instead, systems are needed which are as inconspicuous as possible, such as EarFS, a proprietary development of the Fraunhofer IGD. This is a special ear plug that measures the muscular currents and distortions of the which occur during . The sensor detects even the smallest movements in the face through the way the shape of the ear canal changes and measures muscle currents that arise during the of the face or head.

"The challenge was that these currents and movements are sometimes very small and need to be intensified," explains Denys Matthies, scientist at the Fraunhofer IGD. "In addition, the sensors cannot be interfered with by other movements of the body, such as vibrations during walking or external interferences. To solve this problem, an additional reference electrode was applied to the earlobe which records the signals coming from outside." The signals detected inside the ear are matched with the signals coming from the outside – the remaining useful signal enables clear facial gesture identification, even if the person wearing the EarFS is in motion.

Numerous application and further development possibilities

EarFS not only allows microinteractions with a smartphone, such as accepting and rejecting phone calls or operating the music player. The evaluation of facial movements also allows conclusions about weariness, exertion and other emotional states of the user. As a result, smartphones could warn drivers when signs of weariness and fatigue are registered or could switch automatically to mute when drivers are concentrating attentively. The use of the technology is also imaginable in the medical field. For example, it could help people with locked-in syndrome to communicate more easily by allowing them to operate computers with facial movements. However, this is not the end of the potential for EarFS. "Differential amplification sensing, or, in other words, the amplification of muscle currents and ear canal distortions while simultaneously filtering out external signals, fills a gap in research," says Matthies. "With the technology, we can also read activities at other parts of the body and separate them from external signals: this opens up further possibilities for us, including the complementary operation of machines in Industry 4.0."

Explore further: Battery-free technology brings gesture recognition to all devices

Related Stories

New audio codec enables unprecedented voice call quality

May 30, 2017

Smartphones can do almost everything you want, but their poor voice quality is still a vexing issue. Fraunhofer researchers have helped develop a new codec to banish this problem. Their solution raises voice quality to an ...

Gesture-driven computers

February 29, 2008

It isn’t always easy to communicate with a computer. Two Fraunhofer Institutes will be presenting new possibilities of man-machine interaction at CeBIT in Hanover (Germany) on March 4 through 9. They will demonstrate how ...

Smart driver seat that responds to gestures

August 3, 2015

A multitude of professional drivers suffer from back problems. One cause: driver seats that are inadequately adjusted to each driver's ergonomic needs. Fraunhofer's researchers, in joint collaboration with Isringhausen GmbH ...

Strengthening cybersecurity through research

March 14, 2017

Mobile computing has become a fundamental feature in modern day life as people develop an unprecedented reliance on smart phones and tablets. However, along with their ubiquity comes a host of risks that can affect personal ...

Recommended for you

What can snakes teach us about engineering friction?

May 21, 2018

If you want to know how to make a sneaker with better traction, just ask a snake. That's the theory driving the research of Hisham Abdel-Aal, Ph.D., an associate teaching professor from Drexel University's College of Engineering ...

Flexible, highly efficient multimodal energy harvesting

May 21, 2018

A 10-fold increase in the ability to harvest mechanical and thermal energy over standard piezoelectric composites may be possible using a piezoelectric ceramic foam supported by a flexible polymer support, according to Penn ...


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.