Why we look at the puppet, not the ventriloquist

August 30, 2013 by Steve Hartsoe, Duke University
Why we look at the puppet, not the ventriloquist
Graduate student Daniel Pages is co-author of a paper studying the role sight plays in hearing. Credit: Jennifer Groh

(Medical Xpress)—As ventriloquists have long known, your eyes can sometimes tell your brain where a sound is coming from more convincingly than your ears can.

A series of experiments in humans and monkeys by Duke University researchers has found that the does not require simultaneous visual and audio stimulation to locate the source of a sound. Rather, visual feedback obtained from trying to find a sound with the eyes had a stronger effect than presented at the same time as the audio, according to the Duke study.

The findings could help those with mild learn to localize voices better, improving their ability to communicate in , said Jennifer Groh, a professor of psychology and neuroscience at Duke.

Locating where a sound is coming from is partially learned with the aid of vision. Researchers sought to learn more about how the brain locates the source of a sound when the source is unclear and there are a number of possible visual matches.

"Our study is related to ventriloquism, in which the of a puppet's mouth 'captures' the sound of the puppeteer's voice," Groh said. "It is thought that one reason this illusion occurs is because vision normally teaches the brain how to tell where sounds are coming from. We investigated how the brain knows which visual stimulus should capture the location of a sound, such as why it is the puppet's mouth and not some other visual stimulus."

The study, which appears Thursday (Aug. 29) in the journal PLOS ONE, tested two competing . In one, the brain determines the location of a sound based on the simultaneous occurrence of audio and its visual source. In the other, the brain uses a "guess and check" method. In this scenario, visual feedback sent to the brain after the eye focuses on a sound affects how the eye searches for that sound in the future, possibly through the brain's reward-related .

In both paradigms, the visual stimulus—an LED—was displaced from the sound. Groh's team then looked for evidence that the LED caused a persistent mislocation of the sound.

"Surprisingly, we found that visual feedback exerts the more powerful effect on altering localization of sounds," Groh said. "This suggests that the active behavior of looking at the during a ventriloquism performance plays a role in causing the shift in where you hear the voice."

Participants in the study—11 humans and two rhesus monkeys—shifted their sight to a sound under different visual and audio scenarios.

In one scenario, called the "synchrony-only" task, a visual stimulus appeared at the same time as a sound but too briefly to provide feedback after an eye movement to that sound.

In another, the "feedback-only" task, the visual stimulus appeared during the execution of an eye movement to a sound, but was never on at the same time as the sound.

The study found that the "feedback-only task" exerted a much more powerful effect on the estimation of sound location, as measured with eye tracking, than did the other scenario. This suggests that those who have difficulty localizing sounds may benefit from practice involving eye movements.

On average, participants altered their eye movements in the direction of the lightsâ?? location to a greater degree, about a quarter of the way, when the was presented as feedback than when it was presented at the same time as the sound, the study found.

"This is about the brain's self-improvement skills," said co-author Daniel Pages, a graduate student in Psychology & Neuroscience at Duke. "What we're getting at is how the brain uses different types of information to improve how it does its job. In this case, it uses vision coupled with eye movements to improve hearing."

"We were surprised at how important the were," Groh said. "But finding sounds is really hard. Feedback about your performance is important for anything that is difficult, whether it is the B- you get on your homework or the error your eyes detect in localizing a ."

Explore further: Rewired visual input to sound-processing part of the brain leads to compromised hearing

More information: Pages, D. and Groh, J. Looking at the Ventriloquist: Visual Outcome of eye Movements Calibrates Sound Localization, PLOS ONE, Aug. 29, 2013. DOI: 10.1371/journal.pone.0072562

Related Stories

Rewired visual input to sound-processing part of the brain leads to compromised hearing

August 22, 2012
Scientists at Georgia State University have found that the ability to hear is lessened when, as a result of injury, a region of the brain responsible for processing sounds receives both visual and auditory inputs.

Researchers find humans process echo location and echo suppression differently

August 28, 2013
(Medical Xpress)—A trio of German researchers has found that human beings listening to sounds that have a corresponding echo, process the sounds differently depending on whether they are using echo location or echo suppression. ...

Appearance means more than sound when judging music, research finds

August 20, 2013
(Medical Xpress)—We expect musical experts to focus on sound when judging competitions, and experts believe that their judgments are based on what they hear, not what they see. However, a study by Chia-Jung Tsay of University ...

Reward linked to image is enough to activate brain's visual cortex

March 21, 2013
Once rhesus monkeys learn to associate a picture with a reward, the reward by itself becomes enough to alter the activity in the monkeys' visual cortex. This finding was made by neurophysiologists Wim Vanduffel and John Arsenault ...

Inner speech speaks volumes about the brain

July 16, 2013
Whether you're reading the paper or thinking through your schedule for the day, chances are that you're hearing yourself speak even if you're not saying words out loud. This internal speech—the monologue you "hear" inside ...

Gamblers like noisy slot machines—it reinforces the rewarding feeling after a win

July 2, 2013
Winning sounds on slot machines make gambling more exciting, according to a new study by Mike Dixon and colleagues from the University of Waterloo in Canada. Moreover, their work shows that sounds also cause players to overestimate ...

Recommended for you

Even without nudging blood pressure up, high-salt diet hobbles the brain

January 16, 2018
A high-salt diet may spell trouble for the brain—and for mental performance—even if it doesn't push blood pressure into dangerous territory, new research has found.

Neuroscientists suggest a model for how we gain volitional control of what we hold in our minds

January 16, 2018
Working memory is a sort of "mental sketchpad" that allows you to accomplish everyday tasks such as calling in your hungry family's takeout order and finding the bathroom you were just told "will be the third door on the ...

New study reveals why some people are more creative than others

January 16, 2018
Creativity is often defined as the ability to come up with new and useful ideas. Like intelligence, it can be considered a trait that everyone – not just creative "geniuses" like Picasso and Steve Jobs – possesses in ...

Brain imaging predicts language learning in deaf children

January 15, 2018
In a new international collaborative study between The Chinese University of Hong Kong and Ann & Robert H. Lurie Children's Hospital of Chicago, researchers created a machine learning algorithm that uses brain scans to predict ...

Preterm babies may suffer setbacks in auditory brain development, speech

January 15, 2018
Preterm babies born early in the third trimester of pregnancy are likely to experience delays in the development of the auditory cortex, a brain region essential to hearing and understanding sound, a new study reveals. Such ...

BOLD view of white matter

January 15, 2018
The brain consists of gray matter, which contains the nerve cell bodies (neurons), and white matter, bundles of long nerve fibers (axons) that until recently were considered passive transmitters of signals between different ...


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.