Machine Translates Thoughts into Speech in Real Time

Dec 21, 2009 By Lisa Zyga feature
Model of the brain-machine interface for real-time synthetic speech production. The stroke-induced lesion (red X) disables speech output, but speech motor planning in the cerebral cortex remains intact. Signals collected from an electrode in the speech motor cortex are amplified and sent wirelessly across the scalp as FM radio signals. The Neuralynx System amplifies, converts, and sorts the signals. The neural decoder then translates the signals into speech commands for the speech synthesizer. Credit: Guenther, et al.

(PhysOrg.com) -- By implanting an electrode into the brain of a person with locked-in syndrome, scientists have demonstrated how to wirelessly transmit neural signals to a speech synthesizer. The "thought-to-speech" process takes about 50 milliseconds - the same amount of time for a non-paralyzed, neurologically intact person to speak their thoughts. The study marks the first successful demonstration of a permanently installed, wireless implant for real-time control of an external device.

The study is led by Frank Guenther of the Department of Cognitive and Neural Systems and the Sargent College of Health and Rehabilitation Sciences at Boston University, as well as the Division of Health Science and Technology at Harvard University-Massachusetts Institute of Technology. The research team includes collaborators from Neural Signals, Inc., in Duluth, Georgia; StatsANC LLC in Buenos Aires, Argentina; the Georgia Tech Research Institute in Marietta, Georgia; the Gwinnett Medical Center in Lawrenceville, Georgia; and Emory University Hospital in Atlanta, Georgia. The team published their results in a recent issue of .

“The results of our study show that a brain-machine interface (BMI) user can control sound output directly, rather than having to use a (relatively slow) typing process,” Guenther told PhysOrg.com.

In their study, the researchers tested the technology on a 26-year-old male who had a brain stem at age 16. The brain stem stroke caused a lesion between the volunteer’s that carry out actions and the rest of the brain; while his consciousness and cognitive abilities are intact, he is paralyzed except for slow vertical movement of the eyes. The rare condition is called locked-in syndrome.

Five years ago, when the volunteer was 21 years old, the scientists implanted an electrode near the boundary between the speech-related premotor and primary motor cortex (specifically, the left ventral premotor cortex). Neurites began growing into the electrode and, in three or four months, the neurites produced signaling patterns on the electrode wires that have been maintained indefinitely.

Three years after implantation, the researchers began testing the brain-machine interface for real-time synthetic production. The system is “telemetric” - it requires no wires or connectors passing through the skin, eliminating the risk of infection. Instead, the electrode amplifies and converts neural signals into frequency modulated (FM) radio signals. These signals are wirelessly transmitted across the scalp to two coils, which are attached to the volunteer’s head using a water-soluble paste. The coils act as receiving antenna for the RF signals. The implanted electrode is powered by an induction power supply via a power coil, which is also attached to the head.

The signals are then routed to an electrophysiological recording system that digitizes and sorts them. The sorted spikes, which contain the relevant data, are sent to a neural decoder that runs on a desktop computer. The neural decoder’s output becomes the input to a speech synthesizer, also running on the computer. Finally, the speech synthesizer generates synthetic speech (in the current study, only three vowel sounds were tested). The entire process takes an average of 50 milliseconds.

As the scientists explained, there are no previous electrophysiological studies of neuronal firing in speech motor areas. In order to develop an accurate neural coding scheme, they had to rely on an established neurocomputational model of speech motor control. According to this model, in the left ventral premotor cortex represent intended speech sounds in terms of “formant frequency trajectories.”

In an intact brain, these frequency trajectories are sent to the primary motor cortex where they are transformed into motor commands to the speech articulators. However, in the current study, the researchers had to interpret these frequency trajectories in order to translate them into speech. To do this, the scientists developed a two-dimensional formant frequency space, in which different vowel sounds can be plotted based on two formant frequencies (whose values are represented on the x and y axes).

“The study supported our hypothesis (based on the DIVA model, our neural network model of speech) that the premotor cortex represents intended speech as an ‘auditory trajectory,’ that is, as a set of key frequencies (formant frequencies) that vary with time in the acoustic signal we hear as speech,” Guenther said. “In other words, we could predict the intended sound directly from neural activity in the premotor cortex, rather than try to predict the positions of all the speech articulators individually and then try to reconstruct the intended sound (a much more difficult problem given the small number of neurons from which we recorded). This result provides our first insight into how neurons in the brain represent speech, something that has not been investigated before since there is no animal model for speech.”

To confirm that the neurons in the implanted area were able to carry speech information in the form of formant frequency trajectories, the researchers asked the volunteer to attempt to speak in synchrony with a vowel sequence that was presented auditorily. In later experiments, the volunteer received real-time auditory feedback from the speech synthesizer. During 25 sessions over a five-month period, the volunteer significantly improved the thought-to-speech accuracy. His average hit rate increased from 45% to 70% across sessions, reaching a high of 89% in the last session.

Although the current study focused only on producing a small set of vowels, the researchers think that consonant sounds could be achieved with improvements to the system. While this study used a single three-wire electrode, the use of additional electrodes at multiple recording sites, as well as improved decoding techniques, could lead to rapid, accurate control of a speech synthesizer that could generate a wide range of sounds.

“Our immediate plans involve the implementation of a new synthesizer that can produce consonants as well as vowels but remains simple enough for a BMI user to control,” Guenther said. “We are also working on hardware that will greatly increase the number of neurons that are recorded. We expect to tap into at least 10 times as many neurons in the next implant recipient, which should lead to a dramatic improvement in performance.”

Overall, the work marks a milestone in the development of a permanent neural prosthesis that requires no major external hardware beyond a wireless receiver and laptop computer. Previous brain-machine interfaces for communication applications are very slow, producing only about one word per minute. The new system has the potential to enable real-time conversation, and help minimize the social isolation that accompanies profound paralysis.

Explore further: Researchers reveal pathway that contributes to Alzheimer's disease

More information: Guenther FH, Brumberg JS, Wright EJ, Nieto-Castanon A, Tourville JA, et al. (2009) A Wireless Brain-Machine Interface for Real-Time Speech Synthesis. PLoS ONE 4(12): e8218. doi:10.1371/journal.pone.0008218

4.8 /5 (104 votes)
add to favorites email to friend print save as pdf

Related Stories

Zeroing in on the brain's speech 'receiver'

Jun 20, 2007

A particular resonance pattern in the brain’s auditory processing region appears to be key to its ability to discriminate speech, researchers have found. They found that the inherent rhythm of neural activity called “theta ...

Where the brain makes sense of speech

Dec 19, 2007

Researchers have identified regions of the brain where speech sounds are perceived as having abstract meaning, rather than as just a stream of sensory input. They said their identification of the regions demonstrates that ...

Our faces, not just our ears 'hear' speech: study

Jan 20, 2009

(PhysOrg.com) -- A McGill-led study has found that the perception of speech sounds is modified by stretching facial skin in different directions. Different patterns of skin stretch affect how subjects perceive different words.

Recommended for you

User comments : 27

Adjust slider to filter visible comments by rank

Display comments: newest first

podizzle
5 / 5 (1) Dec 21, 2009
yea buddy! one step closer to controlling my computer with my mind!
el_gramador
5 / 5 (1) Dec 21, 2009
They already have that for a few games here and there. I'm sure a few modifications here and there will allow for something more free and allow for full control with your mind.
NickFun
1 / 5 (1) Dec 21, 2009
It won't be long before Big Brother can simply scan the airwaves and find out who's thinking what. Enjoy what freedom we have left as it will soon become nostalgia!
zevkirsh
5 / 5 (1) Dec 21, 2009
one step closer to telepathy.
the mind is still a black box, theories are how the auditory and speech system may be obscured by plastic adaptation of the neuronal system of the subjects brain to the device that is implanted in it, however, as long as the theory helps the progress and development of the technology, it's utility warrants its explanatory legitimacy, particularly with regard to the mechansim of the action in the artificial speech device (as opposed to natural speech) .
Arikin
5 / 5 (1) Dec 21, 2009
Wonder if an electrode with the ends that fan out gradually to nano scale would help pick up the signals better?

I was imagining a small normal wire in comparison to a nerve cell... Kind of like a skyscraper next to a human. Picks up the group yelling but not the individual.
dan42day
Dec 21, 2009
This comment has been removed by a moderator.
nuge
5 / 5 (2) Dec 22, 2009
That's incredible. Well done, guys. That's the best thing I've heard all month.
mfritz0
5 / 5 (1) Dec 22, 2009
Perhaps I'm a mad scientist, but it seems this procedure could be performed on other species on the planet, such as chimpanzees or dolphins, heck maybe even a horse. Wouldn't that be cool if there was a breakthrough like this to talk with lower species.
gwrede
1 / 5 (1) Dec 22, 2009
scientists implanted an electrode near the boundary between the speech-related premotor and primary motor cortex
I'm not sure why they chose that site. Was it because it was easier to insert the electrode there, or was it because it supported their "auditory trajectory" hypothesis?

I honestly hope it was the former.

*After* the primary motor cortex would have made learning to speak way easier for the patient. Weeks, instead of half a year! Learning to ski, skate, drive, karate, whistle, they're all simply reprogramming the motor cortex. That's what we do. We're actually better at it than any other species.
earls
Dec 22, 2009
This comment has been removed by a moderator.
otto1923
5 / 5 (1) Dec 22, 2009
The internal dialogue... the conversation we are always having with ourselves. We can hear it, we know it exists, and yet there is no evidence that it occurs... yet. The old New Age (?) philo/anthro/huckster? Carlos Castenada said his Yaqui brujo Don Juan Matus taught that the first step on the path to becoming a 'man of power' was learning to turn off the internal dialogue. But we want to be able to broadcast it instead. His methods also included peyote, psilocybin and various other natural herbs and spices; but only in the first book.
thingumbobesquire
3 / 5 (2) Dec 23, 2009
One could imagine with the evolution of this technology, a nano computer system implanted in the brain(or perhaps less invasively, rather as an article of clothing in contact with the electro magnetic brainwave spectrum)such that communication would be possible between parties by thought alone and translated from their native languages automatically. Frankly, this scenario seems no further off than say harnessing matter/anti matter reactions to power inter stellar flight shielding humans from radiation and using inertial force as an equivalent for gravity via effect continuous acceleration deceleration.
otto1923
5 / 5 (2) Dec 23, 2009
@thing
No doubt if telepathy exists in advanced species it would be a natural development of this tech, implanted at first and eventually genetically produced.
NeilFarbstein
5 / 5 (2) Dec 24, 2009
Donovan's brain will be real if this continues.
So will tele-robots that work at the command of a person wearing a thinking cap. It seems that mind reading devices are much closer to reality than anyone thought. The singularity is approaching and its picking up speed.
Duude
2.5 / 5 (2) Dec 25, 2009
What a fascinating development!
However, I can already imagine some of the potential negative consequences if this was to morph into a new form of lie detection, and/or thought police tool as we push into a world of more control.
GREENEMAC
5 / 5 (1) Dec 25, 2009
They should hook this up to Stephen Hawking...give him an easier time in interviews and public speeches.
Nevertheless
not rated yet Dec 25, 2009
The antidote to a malignant 'brave new world' is vigilance in the defense of liberty.
otto1923
not rated yet Dec 26, 2009
new form of lie detection
Lies should be detected. Cheaters should be punished dont you think? Why do people think that the ability to get away with something or to get preferential treatment are unalienable rights? I say cameras on every corner to catch dumbass speeders, tailgaters and red light runners. They restrict my freedom to drive in safety and comfort. Tech will soon make lawyers technically obsolete but people will resist this because they will want to retain the chance to beat the system. This is WRONG.
plasticpower
4 / 5 (1) Dec 27, 2009
Sometimes maybe the system is wrong? I don't feel the speed limit should be 60, I want to drive 65-70 and I am perfectly capable of doing so without putting anyone in any more danger. However, it's the slow drivers who don't look where they're going that cause a lot of the accidents.
otto1923
not rated yet Dec 27, 2009
@plasticpower
See? Like I said- we don't usually need to break laws but we feel it's our right to do so. We WANT to, just as any animal wants to find a way out of the cage. Instinct. I heard 2 lawyers talking yesterday about a simple case with an inevitable conclusion which nevertheless dragged on for years, causing needless hardship and expense, only because the defendent had some influence. Look up the story on the Guess Jeans magnate. Over principle he has ruined lives not to prove any point but because he is an unsettled paranoid. With millions to burn. Enough is enough. Only automated justice is truly blind.
Mark_Holden
not rated yet Dec 27, 2009
Actually, the gov is decades ahead of this. They have been reading ALL of our thoughts for at least 2 decades. Technological Telepathy has long been in use. But of course they just label (like me) people as schizophrenic who say they hear voices or are the victim of gov. mind control programs. Too much for you?
otto1923
not rated yet Dec 27, 2009
What is it- 11.5mhz or something, the frequency emanating from those police repeaters on water towers and which resonates with our brain waves? I read icke too. I doubt if most peoples thoughts are worth reading but they are certainly worth influencing. MK Ultra.
raisin2
not rated yet Dec 28, 2009
This technology is in no way attempting telepathy nor to teach speech. Once areas of the brain are dead they cannot be reanimated (or at least this study is not attempting to do so). The brain distinguishes everything we do. Internal thought is not the same as external verbalization, and the intent and implementation for speech are entirely different. The electrode is placed exactly were it can pick up intents to speak. It could not be placed after the primary motor cortex because that is what is damaged. If the volunteer had impulses going to that area, then he would be speaking, in very rudimentary terms.
bentrider1957
not rated yet Dec 29, 2009
"I would fck the sht out of her!"

"Oh damn, did I just think that outloud?"

"What! STOP IT! ARGH!11"


Anyone remember the Gilligan Island episode where Gilligan eats something and can read the thoughts of others? This story reminds me of that. What happens when we say exactly what we think? "Hey, Ginger, nice t*ts!"
otto1923
not rated yet Dec 30, 2009
She'll have the moral turpitude to thank us for it. Anyways Mary anne was the hot one. *just googled turpitude; maybe I meant maturity or fortitude; maybe knot.
otto1923
not rated yet Dec 30, 2009
That reminds me of the episode of bob Newhart when cousin Larry complimented bobs wife on her very lovely pair of breasts. Out loud.
andyrdj
not rated yet Jan 02, 2010
This will come as a major relief to those poor individuals trapped inside their bodies as they will be able to make their wishes clear - including the wish to end all life support.

I personally think that given that people have the right to end all treatment - effectively choosing a slow a painful death - we should also give them the right to choose a quick and painless death. After all, murderers die by painless lethal injection, so why do we force innocents to suffer more?

I hasten to add that this should be a personal choice - but I know which one I'd make.
clifbank
not rated yet Jan 06, 2010
I'd like to know how to deactivate micro chip brain implants. Since 2001, my famly and I have been harassed continually by the "Big Brother" and the city where I live in here Italy. Even all my co-workers have access to me. I have no privacy. Please send comment by e.mail at clifbank2003@yahoo.com
Than you.
moj85
not rated yet Jan 06, 2010
If the volunteer had impulses going to that area, then he would be speaking, in very rudimentary terms.


Actually, thats not technically true. The impulses to the motor cortex are working fine (this is why putting the electrodes in between the pre-motor and the motor cortexes found signal) but the connections from the motor cortex to the actual MUSCLES was destroyed. This is why he was paralyzed (spinal cord damage).

Interestingly, the signals through your brain to cause movement (in a normal, healthy human) are almost identical to the signals sent while merely THINKING about moving, minus the primary motor cortex (which initiates the movements).