When your eyes override your ears: New insights into the McGurk effect

February 16, 2017
Still frame taken from a video of an actor saying a simple syllable. Humans can use the statistical relationship between what they see in a talker's lips and what they hear in a talker's voice to determine how well they match. The better the match between the seen speech and the heard speech, the more the brain will rely on both cues to create perception. Credit: Magnotti JF and Beauchamp MS, CCBY SA

Seeing is not always believing - visual speech (mouth movements) mismatched with auditory speech (sounds) can result in the perception of an entirely different message. This mysterious illusion is known as the McGurk effect. In new research, published in PLOS Computational Biology, neuroscience researchers have created an algorithm to reveal key insight into why the brain can sometimes muddle up one of the most fundamental aspects of the human experience.

The findings will be useful in understanding patients with deficits and in building computers able to understand auditory and visual speech.

"All humans grow up listening to tens of thousands of speech examples, with the result that our brains contain a comprehensive mapping of the likelihood that any given pair of and speech sounds go together," said Dr. Michael Beauchamp, professor of neurosurgery at Baylor College of Medicine and senior author on the paper with John Magnotti, postdoctoral research fellow at Baylor. "In everyday situations we are frequently confronted with multiple talkers emitting auditory and visual speech cues, and the must decide whether or not to integrate a particular combination of voice and face."

"Even though our senses are constantly bombarded with information, our brain effortlessly selects the verbal and nonverbal speech of our conversation partners from this cacophony," Magnotti said.

The McGurk effect is an example of when this goes wrong. It happens when mouth movements that are seen can override what is heard, causing a person to perceive a different sound than what is actually being said. Only when the eyes are closed, and when the sound is being heard, can the correct message be perceived. For example, the visual "ga" combined with the auditory "ba" results in the perception of "da." (see http://openwetware.org/wiki/Beauchamp:McGurk_CI_Stimuli for examples)

Magnotti and Beauchamp were able to create an algorithm model of multisensory speech perception based on the principle of causal inference, which means given a particular pair of auditory and visual syllables, the brain calculates the likelihood they are from single versus multiple talkers and uses this likelihood to determine the final speech perception.

"We compared our model with an alternative model that is identical, except that it always integrates the available cues, meaning there is no casual inference of speech perception," said Beauchamp, who also is director of the Core for Advanced MRI at Baylor. "Using data from a large number of subjects, the model with causal inference better predicted how humans would or would not integrate audiovisual speech syllables."

"The results suggest a fundamental role for a causal inference type calculation going on in the brain during multisensory speech perception," Magnotti said.

Researchers already have an idea of how and where the brain separately encodes auditory speech and visual speech, but this algorithm shines light on the process of how they are integrated. It will serve as a guide, highlighting specific brain regions that will be essential for multisensory speech perception.

"Understanding how the brain combines information from multiple senses will provide insight into ways to improve declines in speech perception due to typical aging and even to develop devices that could enhance hearing across the life span," Beauchamp said.

Explore further: Hearing with your eyes—a Western style of speech perception

More information: Magnotti JF, Beauchamp MS (2016) A Causal Inference Model Explains Perception of the McGurk Effect and Other Incongruent Audiovisual Speech. PLoS Comput Biol 12(12): e1005229.DOI: 10.1371/journal.pcbi.1005229

Related Stories

Hearing with your eyes—a Western style of speech perception

November 15, 2016
Which parts of a person's face do you look at when you listen them speak? Lip movements affect the perception of voice information from the ears when listening to someone speak, but native Japanese speakers are mostly unaffected ...

Babies need free tongue movement to decipher speech sounds

October 12, 2015
Inhibiting infants' tongue movements impedes their ability to distinguish between speech sounds, researchers with the University of British Columbia have found. The study is the first to discover a direct link between infants' ...

In loud rooms our brains 'hear' in a different way – new findings

May 6, 2016
When we talk face to face, we exchange many more signals than just words. We communicate using our body posture, facial expressions and head and eye movements; but also through the rhythms that are produced when someone is ...

15 years of brain research: Multisensory speech perception examined

December 20, 2012
Research on multisensory speech perception in recent years has helped revolutionize our understanding of how the brain organizes the information it receives from our many different senses, UC Riverside psychology professor ...

Sounds can help develop speech and gestures in children with autism

February 24, 2016
Children with autism and other similar conditions often have difficulties in several areas of communication. A new doctoral thesis in linguistics from the University of Gothenburg shows that these children can develop speech, ...

Recommended for you

Discovery deepens understanding of brain's sensory circuitry

December 12, 2017
Because they provide an exemplary physiological model of how the mammalian brain receives sensory information, neural structures called "mouse whisker barrels" have been the subject of study by neuroscientists around the ...

Stuttering: Stop signals in the brain disturb speech flow

December 12, 2017
One per cent of adults and five per cent of children are unable to achieve what most of us take for granted—speaking fluently. Instead, they struggle with words, often repeating the beginning of a word, for example "G-g-g-g-g-ood ...

Intermittent fasting found to increase cognitive functions in mice

December 12, 2017
(Medical Xpress)—The Daily Mail spoke with the leader of a team of researchers with the National Institute on Aging in the U.S. and reports that they have found that putting mice on a diet consisting of eating nothing every ...

Neuroscientists show deep brain waves occur more often during navigation and memory formation

December 12, 2017
UCLA neuroscientists are the first to show that rhythmic waves in the brain called theta oscillations happen more often when someone is navigating an unfamiliar environment, and that the more quickly a person moves, the more ...

How Zika virus induces congenital microcephaly

December 12, 2017
Epidemiological studies show that in utero fetal infection with the Zika virus (ZIKV) may lead to microcephaly, an irreversible congenital malformation of the brain characterized by an incomplete development of the cerebral ...

Presurgical imaging may predict whether epilepsy surgery will work

December 11, 2017
Surgery to remove a part of the brain to give relief to patients with epilepsy doesn't always result in complete seizure relief, but statisticians at Rice University have developed a method for integrating neuroimaging scans ...

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.