Decoding brain waves to eavesdrop on what we hear
Neuroscientists may one day be able to hear the imagined speech of a patient unable to speak due to stroke or paralysis, according to University of California, Berkeley, researchers.
These scientists have succeeded in decoding electrical activity in the brain's temporal lobe the seat of the auditory system as a person listens to normal conversation. Based on this correlation between sound and brain activity, they then were able to predict the words the person had heard solely from the temporal lobe activity.
"This is huge for patients who have damage to their speech mechanisms because of a stroke or Lou Gehrig's disease and can't speak," said co-author Robert Knight, a UC Berkeley professor of psychology and neuroscience. "If you could eventually reconstruct imagined conversations from brain activity, thousands of people could benefit."
"This research is based on sounds a person actually hears, but to use it for reconstructing imagined conversations, these principles would have to apply to someone's internal verbalizations," cautioned first author Brian N. Pasley, a post-doctoral researcher in the center. "There is some evidence that hearing the sound and imagining the sound activate similar areas of the brain. If you can understand the relationship well enough between the brain recordings and sound, you could either synthesize the actual sound a person is thinking, or just write out the words with a type of interface device."
This video is not supported by your browser at this time.
These are frequency spectrograms of the actual spoken words (top) and the sounds as reconstructed by two separate models based solely on recorded temporal lobe activity in a volunteer subject. The words -- Waldo, structure, doubt and property -- are more or less recognizable, even though the model had never encountered these specific words before. Credit: Brian Pasley, UC BerkeleyIn addition to the potential for expanding the communication ability of the severely disabled, he noted, the research also "is telling us a lot about how the brain in normal people represents and processes speech sounds."
Pasley and his colleagues at UC Berkeley, UC San Francisco, University of Maryland and The Johns Hopkins University report their findings Jan. 31 in the open-access journal PLoS Biology.
Help from epilepsy patients
They enlisted the help of people undergoing brain surgery to determine the location of intractable seizures so that the area can be removed in a second surgery. Neurosurgeons typically cut a hole in the skull and safely place electrodes on the brain surface or cortex in this case, up to 256 electrodes covering the temporal lobe to record activity over a period of a week to pinpoint the seizures. For this study, 15 neurosurgical patients volunteered to participate.
Pasley visited each person in the hospital to record the brain activity detected by the electrodes as they heard 5-10 minutes of conversation. Pasley used this data to reconstruct and play back the sounds the patients heard. He was able to do this because there is evidence that the brain breaks down sound into its component acoustic frequencies for example, between a low of about 1 Hertz (cycles per second) to a high of about 8,000 Hertz that are important for speech sounds.
Pasley tested two different computational models to match spoken sounds to the pattern of activity in the electrodes. The patients then heard a single word, and Pasley used the models to predict the word based on electrode recordings.
"We are looking at which cortical sites are increasing activity at particular acoustic frequencies, and from that, we map back to the sound," Pasley said. He compared the technique to a pianist who knows the sounds of the keys so well that she can look at the keys another pianist is playing in a sound-proof room and "hear" the music, much as Ludwig van Beethoven was able to "hear" his compositions despite being deaf.
The better of the two methods was able to reproduce a sound close enough to the original word for Pasley and his fellow researchers to correctly guess the word.
"We think we would be more accurate with an hour of listening and recording and then repeating the word many times," Pasley said. But because any realistic device would need to accurately identify words heard the first time, he decided to test the models using only a single trial.
"This research is a major step toward understanding what features of speech are represented in the human brain" Knight said. "Brian's analysis can reproduce the sound the patient heard, and you can actually recognize the word, although not at a perfect level."
Knight predicts that this success can be extended to imagined, internal verbalizations, because scientific studies have shown that when people are asked to imagine speaking a word, similar brain regions are activated as when the person actually utters the word.
"With neuroprosthetics, people have shown that it's possible to control movement with brain activity," Knight said. "But that work, while not easy, is relatively simple compared to reconstructing language. This experiment takes that earlier work to a whole new level."
Based on earlier work with ferrets
The current research builds on work by other researchers about how animals encode sounds in the brain's auditory cortex. In fact, some researchers, including the study's coauthors at the University of Maryland, have been able to guess the words ferrets were read by scientists based on recordings from the brain, even though the ferrets were unable to understand the words.
The ultimate goal of the UC Berkeley study was to explore how the human brain encodes speech and determine which aspects of speech are most important for understanding.
"At some point, the brain has to extract away all that auditory information and just map it onto a word, since we can understand speech and words regardless of how they sound," Pasley said. "The big question is, What is the most meaningful unit of speech? A syllable, a phone, a phoneme? We can test these hypotheses using the data we get from these recordings."
More information: Pasley BN, David SV, Mesgarani N, Flinker A, Shamma SA, et al. (2012) Reconstructing Speech from Human Auditory Cortex. PLoS Biol 10(1): e1001251. doi:10.1371/journal.pbio.1001251
Provided by University of California - Berkeley
- Our brains are wired so we can better hear ourselves speak, study shows Dec 08, 2010 | not rated yet | 0
- Researchers identify components of speech recognition pathway in humans Jun 22, 2011 | not rated yet | 0
- Read my lips: Using multiple senses in speech perception (Video) Feb 11, 2009 | not rated yet | 0
- Skilled readers rely on their brain's 'visual dictionary' to recognize words Nov 14, 2011 | not rated yet | 0
- Lend me your ears -- and the world will sound very different Jan 14, 2008 | not rated yet | 0
- Motion perception revisited: High Phi effect challenges established motion perception assumptions Apr 23, 2013 | 3 / 5 (2) | 2
- Anything you can do I can do better: Neuromolecular foundations of the superiority illusion (Update) Apr 02, 2013 | 4.5 / 5 (11) | 5
- The visual system as economist: Neural resource allocation in visual adaptation Mar 30, 2013 | 5 / 5 (2) | 9
- Separate lives: Neuronal and organismal lifespans decoupled Mar 27, 2013 | 4.9 / 5 (8) | 0
- Sizing things up: The evolutionary neurobiology of scale invariance Feb 28, 2013 | 4.8 / 5 (10) | 14
Pressure-volume curve: Elastic Recoil Pressure don't make sense
19 hours ago From pressure-volume curve of the lung and chest wall (attached photo), I don't understand why would the elastic recoil pressure of the lung is...
If you became brain-dead, would you want them to pull the plug?
May 17, 2013 I'd want the rest of me to stay alive. Sure it's a lousy way to live but it beats being all-the-way dead. Maybe if I make it 20 years they'll...
MRI bill question
May 15, 2013 Dear PFers, The hospital gave us a $12k bill for one MRI (head with contrast). The people I talked to at the hospital tell me that they do not...
Ratio of Hydrogen of Oxygen in Dessicated Animal Protein
May 13, 2013 As an experiment, for the past few months I've been consuming at least one portion of Jell-O or unflavored Knox gelatin per day. I'm 64, in very...
Alcohol and acetaminophen
May 13, 2013 Edit: sorry for the typo in the title , can't edit I looked around on google quite a bit and it's very hard to find precise information on the...
Marie Curie's leukemia
May 13, 2013 Does anyone know what might be the cause of Marie Curie's cancer
- More from Physics Forums - Medical Sciences
More news stories
For combat veterans suffering from post-traumatic stress disorder, 'fear circuitry' in the brain never rests
Chronic trauma can inflict lasting damage to brain regions associated with fear and anxiety. Previous imaging studies of people with post-traumatic stress disorder, or PTSD, have shown that these brain regions can over-or ...
Neuroscience 21 hours ago | 5 / 5 (1) | 0 |
The neural machinery underlying our olfactory sense continues to be an enigma for neuroscience. A recent review in Neuron seeks to expand traditional ideas about how neurons in the olfactory bulb might encode information about ...
Neuroscience May 17, 2013 | not rated yet | 0 |
(Medical Xpress)—What if the quality of your work depends more on your focus on the piano keys or canvas or laptop than your musical or painting or computing skills? If target users can be convinced, they ...
Neuroscience May 17, 2013 | 3.7 / 5 (3) | 0 |
Neurological disorders can have a devastating impact on the lives of sufferers and their families.
Neuroscience May 17, 2013 | 5 / 5 (1) | 0 |
If you're a left-brain thinker, chances are you use your right hand to hold your cell phone up to your right ear, according to a newly published study from Henry Ford Hospital in Detroit.
Neuroscience May 16, 2013 | 2 / 5 (2) | 0 |
An increasing number of U.S. children are experiencing gastrointestinal issues that require interventions to resolve, according to research presented at Digestive Disease Week (DDW).
12 hours ago | not rated yet | 0 |
The latest makeover to a massive psychiatric tome honored by some, reviled by others and even called the "Bible" of mental disorders is being released Saturday with a host of new changes.
9 hours ago | not rated yet | 0
A new case of the deadly coronavirus has been detected in Saudi Arabia where 15 people have already died after contracting it, the health ministry announced on Saturday on its Internet website.
10 hours ago | not rated yet | 0
Big names in medicine are set to give an upbeat assessment of the war on AIDS on Tuesday, 30 years after French researchers identified the virus that causes the disease.
20 hours ago | 5 / 5 (1) | 0
A ground-breaking advance in colonoscopy technology signals the future of colorectal care, according to research presented today at Digestive Disease Week(DDW). Additional research focuses on optimizing the minimal withdrawal ...
12 hours ago | 5 / 5 (2) | 0