Machine Translates Thoughts into Speech in Real Time

Machine Translates Thoughts into Speech in Real Time
Model of the brain-machine interface for real-time synthetic speech production. The stroke-induced lesion (red X) disables speech output, but speech motor planning in the cerebral cortex remains intact. Signals collected from an electrode in the speech motor cortex are amplified and sent wirelessly across the scalp as FM radio signals. The Neuralynx System amplifies, converts, and sorts the signals. The neural decoder then translates the signals into speech commands for the speech synthesizer. Credit: Guenther, et al.

( -- By implanting an electrode into the brain of a person with locked-in syndrome, scientists have demonstrated how to wirelessly transmit neural signals to a speech synthesizer. The "thought-to-speech" process takes about 50 milliseconds - the same amount of time for a non-paralyzed, neurologically intact person to speak their thoughts. The study marks the first successful demonstration of a permanently installed, wireless implant for real-time control of an external device.

The study is led by Frank Guenther of the Department of Cognitive and Neural Systems and the Sargent College of Health and Rehabilitation Sciences at Boston University, as well as the Division of Health Science and Technology at Harvard University-Massachusetts Institute of Technology. The research team includes collaborators from Neural Signals, Inc., in Duluth, Georgia; StatsANC LLC in Buenos Aires, Argentina; the Georgia Tech Research Institute in Marietta, Georgia; the Gwinnett Medical Center in Lawrenceville, Georgia; and Emory University Hospital in Atlanta, Georgia. The team published their results in a recent issue of .

“The results of our study show that a brain-machine interface (BMI) user can control sound output directly, rather than having to use a (relatively slow) typing process,” Guenther told

In their study, the researchers tested the technology on a 26-year-old male who had a brain stem at age 16. The brain stem stroke caused a lesion between the volunteer’s that carry out actions and the rest of the brain; while his consciousness and cognitive abilities are intact, he is paralyzed except for slow vertical movement of the eyes. The rare condition is called locked-in syndrome.

Five years ago, when the volunteer was 21 years old, the scientists implanted an electrode near the boundary between the speech-related premotor and primary motor cortex (specifically, the left ventral premotor cortex). Neurites began growing into the electrode and, in three or four months, the neurites produced signaling patterns on the electrode wires that have been maintained indefinitely.

Three years after implantation, the researchers began testing the brain-machine interface for real-time synthetic production. The system is “telemetric” - it requires no wires or connectors passing through the skin, eliminating the risk of infection. Instead, the electrode amplifies and converts neural signals into frequency modulated (FM) radio signals. These signals are wirelessly transmitted across the scalp to two coils, which are attached to the volunteer’s head using a water-soluble paste. The coils act as receiving antenna for the RF signals. The implanted electrode is powered by an induction power supply via a power coil, which is also attached to the head.

The signals are then routed to an electrophysiological recording system that digitizes and sorts them. The sorted spikes, which contain the relevant data, are sent to a neural decoder that runs on a desktop computer. The neural decoder’s output becomes the input to a speech synthesizer, also running on the computer. Finally, the speech synthesizer generates synthetic speech (in the current study, only three vowel sounds were tested). The entire process takes an average of 50 milliseconds.

As the scientists explained, there are no previous electrophysiological studies of neuronal firing in speech motor areas. In order to develop an accurate neural coding scheme, they had to rely on an established neurocomputational model of speech motor control. According to this model, in the left ventral premotor cortex represent intended speech sounds in terms of “formant frequency trajectories.”

In an intact brain, these frequency trajectories are sent to the primary motor cortex where they are transformed into motor commands to the speech articulators. However, in the current study, the researchers had to interpret these frequency trajectories in order to translate them into speech. To do this, the scientists developed a two-dimensional formant frequency space, in which different vowel sounds can be plotted based on two formant frequencies (whose values are represented on the x and y axes).

“The study supported our hypothesis (based on the DIVA model, our neural network model of speech) that the premotor cortex represents intended speech as an ‘auditory trajectory,’ that is, as a set of key frequencies (formant frequencies) that vary with time in the acoustic signal we hear as speech,” Guenther said. “In other words, we could predict the intended sound directly from neural activity in the premotor cortex, rather than try to predict the positions of all the speech articulators individually and then try to reconstruct the intended sound (a much more difficult problem given the small number of neurons from which we recorded). This result provides our first insight into how neurons in the brain represent speech, something that has not been investigated before since there is no animal model for speech.”

To confirm that the neurons in the implanted area were able to carry speech information in the form of formant frequency trajectories, the researchers asked the volunteer to attempt to speak in synchrony with a vowel sequence that was presented auditorily. In later experiments, the volunteer received real-time auditory feedback from the speech synthesizer. During 25 sessions over a five-month period, the volunteer significantly improved the thought-to-speech accuracy. His average hit rate increased from 45% to 70% across sessions, reaching a high of 89% in the last session.

Although the current study focused only on producing a small set of vowels, the researchers think that consonant sounds could be achieved with improvements to the system. While this study used a single three-wire electrode, the use of additional electrodes at multiple recording sites, as well as improved decoding techniques, could lead to rapid, accurate control of a speech synthesizer that could generate a wide range of sounds.

“Our immediate plans involve the implementation of a new synthesizer that can produce consonants as well as vowels but remains simple enough for a BMI user to control,” Guenther said. “We are also working on hardware that will greatly increase the number of neurons that are recorded. We expect to tap into at least 10 times as many neurons in the next implant recipient, which should lead to a dramatic improvement in performance.”

Overall, the work marks a milestone in the development of a permanent neural prosthesis that requires no major external hardware beyond a wireless receiver and laptop computer. Previous brain-machine interfaces for communication applications are very slow, producing only about one word per minute. The new system has the potential to enable real-time conversation, and help minimize the social isolation that accompanies profound paralysis.

More information: Guenther FH, Brumberg JS, Wright EJ, Nieto-Castanon A, Tourville JA, et al. (2009) A Wireless Brain-Machine Interface for Real-Time Speech Synthesis. PLoS ONE 4(12): e8218. doi:10.1371/journal.pone.0008218

Copyright 2009
All rights reserved. This material may not be published, broadcast, rewritten or redistributed in whole or part without the express written permission of

Citation: Machine Translates Thoughts into Speech in Real Time (2009, December 21) retrieved 14 October 2019 from
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.

Feedback to editors

User comments

Dec 21, 2009
yea buddy! one step closer to controlling my computer with my mind!

Dec 21, 2009
They already have that for a few games here and there. I'm sure a few modifications here and there will allow for something more free and allow for full control with your mind.

Dec 21, 2009
It won't be long before Big Brother can simply scan the airwaves and find out who's thinking what. Enjoy what freedom we have left as it will soon become nostalgia!

Dec 21, 2009
This comment has been removed by a moderator.

Dec 22, 2009
That's incredible. Well done, guys. That's the best thing I've heard all month.

Dec 22, 2009
Perhaps I'm a mad scientist, but it seems this procedure could be performed on other species on the planet, such as chimpanzees or dolphins, heck maybe even a horse. Wouldn't that be cool if there was a breakthrough like this to talk with lower species.

Dec 22, 2009
scientists implanted an electrode near the boundary between the speech-related premotor and primary motor cortex
I'm not sure why they chose that site. Was it because it was easier to insert the electrode there, or was it because it supported their "auditory trajectory" hypothesis?

I honestly hope it was the former.

*After* the primary motor cortex would have made learning to speak way easier for the patient. Weeks, instead of half a year! Learning to ski, skate, drive, karate, whistle, they're all simply reprogramming the motor cortex. That's what we do. We're actually better at it than any other species.

Dec 22, 2009
This comment has been removed by a moderator.

Dec 23, 2009
One could imagine with the evolution of this technology, a nano computer system implanted in the brain(or perhaps less invasively, rather as an article of clothing in contact with the electro magnetic brainwave spectrum)such that communication would be possible between parties by thought alone and translated from their native languages automatically. Frankly, this scenario seems no further off than say harnessing matter/anti matter reactions to power inter stellar flight shielding humans from radiation and using inertial force as an equivalent for gravity via effect continuous acceleration deceleration.

Dec 24, 2009
Donovan's brain will be real if this continues.
So will tele-robots that work at the command of a person wearing a thinking cap. It seems that mind reading devices are much closer to reality than anyone thought. The singularity is approaching and its picking up speed.

Dec 25, 2009
What a fascinating development!
However, I can already imagine some of the potential negative consequences if this was to morph into a new form of lie detection, and/or thought police tool as we push into a world of more control.

Dec 25, 2009
They should hook this up to Stephen Hawking...give him an easier time in interviews and public speeches.

Dec 25, 2009
The antidote to a malignant 'brave new world' is vigilance in the defense of liberty.

Dec 27, 2009
Sometimes maybe the system is wrong? I don't feel the speed limit should be 60, I want to drive 65-70 and I am perfectly capable of doing so without putting anyone in any more danger. However, it's the slow drivers who don't look where they're going that cause a lot of the accidents.

Dec 27, 2009
Actually, the gov is decades ahead of this. They have been reading ALL of our thoughts for at least 2 decades. Technological Telepathy has long been in use. But of course they just label (like me) people as schizophrenic who say they hear voices or are the victim of gov. mind control programs. Too much for you?

Dec 28, 2009
This technology is in no way attempting telepathy nor to teach speech. Once areas of the brain are dead they cannot be reanimated (or at least this study is not attempting to do so). The brain distinguishes everything we do. Internal thought is not the same as external verbalization, and the intent and implementation for speech are entirely different. The electrode is placed exactly were it can pick up intents to speak. It could not be placed after the primary motor cortex because that is what is damaged. If the volunteer had impulses going to that area, then he would be speaking, in very rudimentary terms.

Jan 02, 2010
This will come as a major relief to those poor individuals trapped inside their bodies as they will be able to make their wishes clear - including the wish to end all life support.

I personally think that given that people have the right to end all treatment - effectively choosing a slow a painful death - we should also give them the right to choose a quick and painless death. After all, murderers die by painless lethal injection, so why do we force innocents to suffer more?

I hasten to add that this should be a personal choice - but I know which one I'd make.

Jan 06, 2010
I'd like to know how to deactivate micro chip brain implants. Since 2001, my famly and I have been harassed continually by the "Big Brother" and the city where I live in here Italy. Even all my co-workers have access to me. I have no privacy. Please send comment by e.mail at
Than you.

Jan 06, 2010
If the volunteer had impulses going to that area, then he would be speaking, in very rudimentary terms.

Actually, thats not technically true. The impulses to the motor cortex are working fine (this is why putting the electrodes in between the pre-motor and the motor cortexes found signal) but the connections from the motor cortex to the actual MUSCLES was destroyed. This is why he was paralyzed (spinal cord damage).

Interestingly, the signals through your brain to cause movement (in a normal, healthy human) are almost identical to the signals sent while merely THINKING about moving, minus the primary motor cortex (which initiates the movements).

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more