Why we look at the puppet, not the ventriloquist

August 30, 2013 by Steve Hartsoe
Why we look at the puppet, not the ventriloquist
Graduate student Daniel Pages is co-author of a paper studying the role sight plays in hearing. Credit: Jennifer Groh

(Medical Xpress)—As ventriloquists have long known, your eyes can sometimes tell your brain where a sound is coming from more convincingly than your ears can.

A series of experiments in humans and monkeys by Duke University researchers has found that the does not require simultaneous visual and audio stimulation to locate the source of a sound. Rather, visual feedback obtained from trying to find a sound with the eyes had a stronger effect than presented at the same time as the audio, according to the Duke study.

The findings could help those with mild learn to localize voices better, improving their ability to communicate in , said Jennifer Groh, a professor of psychology and neuroscience at Duke.

Locating where a sound is coming from is partially learned with the aid of vision. Researchers sought to learn more about how the brain locates the source of a sound when the source is unclear and there are a number of possible visual matches.

"Our study is related to ventriloquism, in which the of a puppet's mouth 'captures' the sound of the puppeteer's voice," Groh said. "It is thought that one reason this illusion occurs is because vision normally teaches the brain how to tell where sounds are coming from. We investigated how the brain knows which visual stimulus should capture the location of a sound, such as why it is the puppet's mouth and not some other visual stimulus."

The study, which appears Thursday (Aug. 29) in the journal PLOS ONE, tested two competing . In one, the brain determines the location of a sound based on the simultaneous occurrence of audio and its visual source. In the other, the brain uses a "guess and check" method. In this scenario, visual feedback sent to the brain after the eye focuses on a sound affects how the eye searches for that sound in the future, possibly through the brain's reward-related .

In both paradigms, the visual stimulus—an LED—was displaced from the sound. Groh's team then looked for evidence that the LED caused a persistent mislocation of the sound.

"Surprisingly, we found that visual feedback exerts the more powerful effect on altering localization of sounds," Groh said. "This suggests that the active behavior of looking at the during a ventriloquism performance plays a role in causing the shift in where you hear the voice."

Participants in the study—11 humans and two rhesus monkeys—shifted their sight to a sound under different visual and audio scenarios.

In one scenario, called the "synchrony-only" task, a visual stimulus appeared at the same time as a sound but too briefly to provide feedback after an eye movement to that sound.

In another, the "feedback-only" task, the visual stimulus appeared during the execution of an eye movement to a sound, but was never on at the same time as the sound.

The study found that the "feedback-only task" exerted a much more powerful effect on the estimation of sound location, as measured with eye tracking, than did the other scenario. This suggests that those who have difficulty localizing sounds may benefit from practice involving eye movements.

On average, participants altered their eye movements in the direction of the lightsâ?? location to a greater degree, about a quarter of the way, when the was presented as feedback than when it was presented at the same time as the sound, the study found.

"This is about the brain's self-improvement skills," said co-author Daniel Pages, a graduate student in Psychology & Neuroscience at Duke. "What we're getting at is how the brain uses different types of information to improve how it does its job. In this case, it uses vision coupled with eye movements to improve hearing."

"We were surprised at how important the were," Groh said. "But finding sounds is really hard. Feedback about your performance is important for anything that is difficult, whether it is the B- you get on your homework or the error your eyes detect in localizing a ."

Explore further: Rewired visual input to sound-processing part of the brain leads to compromised hearing

More information: Pages, D. and Groh, J. Looking at the Ventriloquist: Visual Outcome of eye Movements Calibrates Sound Localization, PLOS ONE, Aug. 29, 2013. DOI: 10.1371/journal.pone.0072562

Related Stories

Rewired visual input to sound-processing part of the brain leads to compromised hearing

August 22, 2012
Scientists at Georgia State University have found that the ability to hear is lessened when, as a result of injury, a region of the brain responsible for processing sounds receives both visual and auditory inputs.

Researchers find humans process echo location and echo suppression differently

August 28, 2013
(Medical Xpress)—A trio of German researchers has found that human beings listening to sounds that have a corresponding echo, process the sounds differently depending on whether they are using echo location or echo suppression. ...

Appearance means more than sound when judging music, research finds

August 20, 2013
(Medical Xpress)—We expect musical experts to focus on sound when judging competitions, and experts believe that their judgments are based on what they hear, not what they see. However, a study by Chia-Jung Tsay of University ...

Reward linked to image is enough to activate brain's visual cortex

March 21, 2013
Once rhesus monkeys learn to associate a picture with a reward, the reward by itself becomes enough to alter the activity in the monkeys' visual cortex. This finding was made by neurophysiologists Wim Vanduffel and John Arsenault ...

Inner speech speaks volumes about the brain

July 16, 2013
Whether you're reading the paper or thinking through your schedule for the day, chances are that you're hearing yourself speak even if you're not saying words out loud. This internal speech—the monologue you "hear" inside ...

Gamblers like noisy slot machines—it reinforces the rewarding feeling after a win

July 2, 2013
Winning sounds on slot machines make gambling more exciting, according to a new study by Mike Dixon and colleagues from the University of Waterloo in Canada. Moreover, their work shows that sounds also cause players to overestimate ...

Recommended for you

The neural codes for body movements

July 21, 2017
A small patch of neurons in the brain can encode the movements of many body parts, according to researchers in the laboratory of Caltech's Richard Andersen, James G. Boswell Professor of Neuroscience, Tianqiao and Chrissy ...

Faulty support cells disrupt communication in brains of people with schizophrenia

July 20, 2017
New research has identified the culprit behind the wiring problems in the brains of people with schizophrenia. When researchers transplanted human brain cells generated from individuals diagnosed with childhood-onset schizophrenia ...

Scientists reveal how patterns of brain activity direct specific body movements

July 20, 2017
New research by Columbia scientists offers fresh insight into how the brain tells the body to move, from simple behaviors like walking, to trained movements that may take years to master. The discovery in mice advances knowledge ...

Scientists discover combined sensory map for heat, humidity in fly brain

July 20, 2017
Northwestern University neuroscientists now can visualize how fruit flies sense and process humidity and temperature together through a "sensory map" within their brains, according to new research.

Team traces masculinization in mice to estrogen receptor in inhibitory neurons

July 20, 2017
Researchers at Cold Spring Harbor Laboratory (CSHL) have opened a black box in the brain whose contents explain one of the remarkable yet mysterious facts of life.

Speech language therapy delivered through the Internet leads to similar improvements as in-person treatment

July 20, 2017
Telerehabilitation helps healthcare professionals reach more patients in need, but some worry it doesn't offer the same quality of care as in-person treatment. This isn't the case, according to recent research by Baycrest.


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.