This article has been reviewed according to Science X's editorial process and policies. Editors have highlighted the following attributes while ensuring the content's credibility:


peer-reviewed publication

trusted source


Researchers develop neural decoding that can give back lost speech

Researchers develop neural decoding that can give back lost speech
The proposed neural speech decoding framework. Credit: Nature Machine Intelligence (2024). DOI: 10.1038/s42256-024-00824-8

Losing the ability to speak due to neurological damage can be incredibly isolating. But thanks to recent advancements in technology, there's hope on the horizon. Scientists have been working on neural speech prostheses, special devices that can help people who have trouble speaking by translating brain activity into speech.

In a recent study published in Nature Machine Intelligence, a team of NYU researchers led by Yao Wang—Professor of Electrical and Computer Engineering and Biomedical Engineering at NYU Tandon, as well as a member of NYU WIRELESS—and Adeen Flinker—Associate Professor of Biomedical Engineering at NYU Tandon and Neurology at NYU Grossman School of Medicine—and Tandon ECE Ph.D. student Xupeng Chen presented a significant advancement in the decoding of speech using neural architectures.

They recorded signals from the brain and transformed them into audible speech. Building upon previous research, their work introduces modifications that enhance decoding accuracy across a broader range of voices.

One key innovation lies in the adaptation of neural architectures to accommodate diverse speech patterns. Recent strides in and Brain-Computer Interface (BCI) systems have propelled the development of neural speech prostheses, offering hope to those affected by speech impairments.

One effective method for gathering data to develop such prostheses involves Electrocorticographic (ECoG) recordings obtained from epilepsy surgery patients. Implanted electrodes provide a rare opportunity to collect cortical data during speech with high precision, leading to promising results in speech decoding.

Previously validated on five patients, their updated approach now has been validated over 48 individuals—an order of magnitude larger than in other similar work, ensuring a more robust and generalized decoding process.

Two significant challenges persist in decoding speech from neural signals. First, the limited duration of training data contrasts with the extensive data required for deep learning models. Second, speech production variability, encompassing rate, intonation, and pitch variations, complicates model representation.

The NYU team's approach uses a unique speech synthesizer developed in their previous research. This synthesizer translates a series of interpretable speech "parameters" including pitch, frequency, loudness, etc., into natural sounding speech. The developed system leverages neural network architectures to decode neural signals into speech parameters which the synthesizer uses to produce the intended speech.

The team developed an efficient neural network training pipeline that works effectively with limited training data, and compared the efficacy of different neural network architectures. The system can produce speech that is much closer to the actual voice of the study participants—a unique feature of this approach.

Credit: NYU Tandon School of Engineering

Perhaps most intriguing is the discovery regarding the 's contribution to speech decoding. Traditionally the right hemisphere is overshadowed by the left hemisphere, which is predominantly associated with language functions. However, some of the participants only had electrodes implanted on their right hemisphere, providing the researchers no information about the left-hemisphere's activities. Crucially, they were still able to use the information from the right hemisphere to produce accurate speech decoding.

Not only does this reveal how speech is processed and produced by the brain across the two hemispheres, it also opens new possibilities for therapeutic interventions, particularly in addressing speech disorders like aphasia, following damage to the left hemisphere.

In addition to its scientific findings, the study offers an open-source neuro-decoding pipeline, facilitating collaboration and replication of results within the research community. This initiative promotes transparency and accelerates progress in the field of neural decoding.

The research's implications for understanding the complexities of speech processing and potential therapeutic avenues loom large, and marks a significant milestone in unraveling the mysteries of the human mind's linguistic capabilities, paving the way for future breakthroughs in neuroengineering and clinical interventions.

More information: Xupeng Chen et al, A neural speech decoding framework leveraging deep learning and speech synthesis, Nature Machine Intelligence (2024). DOI: 10.1038/s42256-024-00824-8

Journal information: Nature Machine Intelligence
Citation: Researchers develop neural decoding that can give back lost speech (2024, April 8) retrieved 21 May 2024 from
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.

Explore further

Researchers reconstruct speech from brain activity, illuminating complex neural processes


Feedback to editors