Speech recognition from brain activity

June 16, 2015, Karlsruhe Institute of Technology
Speech recognition from brain activity
Brain activity recorded by electrocorticography (blue circles). From the activity patterns (blue/yellow), spoken words can be recognized. Credit: CSL/KIT

Speech is produced in the human cerebral cortex. Brain waves associated with speech processes can be directly recorded with electrodes located on the surface of the cortex. It has now been shown for the first time that is possible to reconstruct basic units, words, and complete sentences of continuous speech from these brain waves and to generate the corresponding text. Researchers at KIT and Wadsworth Center, USA present their "Brain-to-Text" system in the scientific journal Frontiers in Neuroscience.

"It has long been speculated whether humans may communicate with machines via activity alone," says Tanja Schultz, who conducted the present study with her team at the Cognitive Systems Lab of KIT. "As a major step in this direction, our recent results indicate that both single units in terms of as well as continuously spoken sentences can be recognized from brain activity."

These results were obtained by an interdisciplinary collaboration of researchers of informatics, neuroscience, and medicine. In Karlsruhe, the methods for signal processing and have been developed and applied. "In addition to the decoding of from brain activity, our models allow for a detailed analysis of the brain areas involved in speech processes and their interaction," outline Christian Herff und Dominic Heger, who developed the Brain-to-Text system within their doctoral studies.

The present work is the first that decodes continuously spoken speech and transforms it into a textual representation. For this purpose, cortical information is combined with linguistic knowledge and machine learning algorithms to extract the most likely word sequence. Currently, Brain-to-Text is based on audible speech. However, the results are an important first step for recognizing speech from thought alone.

The was recorded in the USA from 7 epileptic patients, who participated voluntarily in the study during their clinical treatments. An electrode array was placed on the surface of the cerebral cortex (electrocorticography (ECoG)) for their neurological treatment. While patients read aloud sample texts, the ECoG signals were recorded with high resolution in time and space. Later on, the researchers in Karlsruhe analyzed the data to develop Brain-to-Text. In addition to basic science and a better understanding of the highly complex speech processes in the brain, Brain-to-Text might be a building block to develop a means of speech communication for locked-in patients in the future.

Explore further: Study: Speech processing requires both sides of our brain

More information: "Brain-to-text: decoding spoken phrases from phone representations in the brain." Front. Neurosci. 9:217. DOI: 10.3389/fnins.2015.00217

Related Stories

Study: Speech processing requires both sides of our brain

January 15, 2014
We use both sides of our brain for speech, a finding by researchers at New York University and NYU Langone Medical Center that alters previous conceptions about neurological activity. The results, which appear in the journal ...

Syllables that oscillate in neuronal circuits: What neuroscience can say about speech processing in the brain

June 10, 2015
Speech, emitted or received, produces an electrical activity in neurons that neuroscientists measure in the form of "cortical oscillations". To understand speech, as for other cognitive or sensory processes, the brain breaks ...

Researchers find brain area that integrates speech's rhythms

May 18, 2015
Duke and MIT scientists have discovered an area of the brain that is sensitive to the timing of speech, a crucial element of spoken language.

Brain's iconic seat of speech goes silent when we actually talk

February 17, 2015
For 150 years, the iconic Broca's area of the brain has been recognized as the command center for human speech, including vocalization. Now, scientists at UC Berkeley and Johns Hopkins University in Maryland are challenging ...

Finding thoughts in speech

June 23, 2014
For the first time, neuroscientists were able to find out how different thoughts are reflected in neuronal activity during natural conversations. Johanna Derix, Olga Iljina and the interdisciplinary team of Dr. Tonio Ball ...

Recommended for you

New neurons in the adult brain are involved in sensory learning

February 23, 2018
Although we have known for several years that the adult brain can produce new neurons, many questions about the properties conferred by these adult-born neurons were left unanswered. What advantages could they offer that ...

Neuroscientists discover a brain signal that indicates whether speech has been understood

February 22, 2018
Neuroscientists from Trinity College Dublin and the University of Rochester have identified a specific brain signal associated with the conversion of speech into understanding. The signal is present when the listener has ...

Study in mice suggests personalized stem cell treatment may offer relief for multiple sclerosis

February 22, 2018
Scientists have shown in mice that skin cells re-programmed into brain stem cells, transplanted into the central nervous system, help reduce inflammation and may be able to help repair damage caused by multiple sclerosis ...

Nolan film 'Memento' reveals how the brain remembers and interprets events from clues

February 22, 2018
Key repeating moments in the film give viewers the information they need to understand the storyline. The scenes cause identical reactions in the viewer's brain. The results deepen our understanding of how the brain functions, ...

Biomarker, clues to possible therapy found in novel childhood neurogenetic disease

February 22, 2018
Researchers studying a rare genetic disorder that causes severe, progressive neurological problems in childhood have discovered insights into biological mechanisms that drive the disease, along with early clues that an amino ...

A look at the space between mouse brain cells

February 22, 2018
Between the brain's neurons and glial cells is a critical but understudied structure that's been called neuroscience's final frontier: the extracellular space. With a new imaging paradigm, scientists can now see into and ...

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.