'Duet of 1' possible with hand-controlled voice synthesizer

Feb 19, 2012

New technology at the University of British Columbia makes it possible for a person to speak or sing just by using their hands to control a speech synthesizer.

UBC researcher Sidney Fels says the gesture-to-voice-synthesizer technology mirrors processes that human use when they control their own vocal apparatus.

"It's like playing a that plays voice. Applications could include new forms of musical expression and aids for people with speaking disabilities," says Fels, professor of electrical and computer engineering at the Faculty of Applied Science and director of the Media and Graphics Interdisciplinary Centre (MAGIC).

Fels presented the technology today at the annual meeting of the in Vancouver.

Fels and his team used special gloves equipped with 3-D position sensors that locate the hand in space. Certain glove postures are associated with certain areas in the audio spectrum.

The right-hand glove has sensors to detect bending so when a user closes her hand, it creates consonant sounds. Opening the right hand produces in the same fashion as a does when the tongue moves. The left glove controls stop sounds – like the consonant 'B'.

The researchers developed a set collection of gestures that are mapped to consonant sounds. The right glove controls vowels by its location in space horizontally and also controls pitch by its location in space vertically.

"Other possible applications for this discovery are interfaces to make certain tasks easier such as controlling cranes or other heavy machinery," says Fels, whose research interests include human-computer interaction, biomechanical modeling of the upper airway, speech synthesis, and neural networks.

Co-investigators for this project are UBC School of Music Asst. Prof. Robert Pritchard and Johnty Wang, a UBC electrical and masters student and concert pianist.

To date, there have been seven international performances with musicians playing a set of pieces written specifically for the expressive capacities of this particular instrument. "It takes about 100 hours for a performer to learn how to speak and use the system," says Fels.

Explore further: Data-driven audience targeting expands to drone tests in LA

Related Stories

Adults with dyslexia have problems with non-speech sounds too

Jun 01, 2010

(PhysOrg.com) -- Dyslexia is usually associated with persistent reading, spelling, and sometimes speech difficulties that are hard to overcome. One theory proposed to explain the condition is that people with dyslexia suffer ...

Recommended for you

Neuroscientist takes scientific look at art of filmmaking

Feb 24, 2015

Why do so many of us cry at the movies? Why do we flinch when Rocky Balboa takes a punch, duck when the jet careens toward the tower in "Airplane," and tap our toes to the dance numbers in "Chicago" or "Moulin ...

Driverless shuttle will be on the move in UK

Feb 22, 2015

(Phys.org) —"Autonomous public transport" is on the minds of planners who envision self-driving vehicles that would cross over short distances, suited for airport transport, industrial sites, theme parks ...

Challenges for doctors using fitness trackers and apps

Feb 20, 2015

More hospitals and doctors are starting to use data from fitness trackers and health apps to help treat patients. But they are moving cautiously. The technology has a lot of potential, but there are key ch ...

Eye tracking is the next frontier of human-computer interaction

Feb 20, 2015

Eye tracking devices sound a lot more like expensive pieces of scientific research equipment than joysticks – yet if the latest announcements about the latest Assassin's Creed game are anything to go by, eye tracking will become a commonplace ...

User comments : 0

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.