Why we look at the puppet, not the ventriloquist

by Steve Hartsoe
Why we look at the puppet, not the ventriloquist
Graduate student Daniel Pages is co-author of a paper studying the role sight plays in hearing. Credit: Jennifer Groh

(Medical Xpress)—As ventriloquists have long known, your eyes can sometimes tell your brain where a sound is coming from more convincingly than your ears can.

A series of experiments in humans and monkeys by Duke University researchers has found that the does not require simultaneous visual and audio stimulation to locate the source of a sound. Rather, visual feedback obtained from trying to find a sound with the eyes had a stronger effect than presented at the same time as the audio, according to the Duke study.

The findings could help those with mild learn to localize voices better, improving their ability to communicate in , said Jennifer Groh, a professor of psychology and neuroscience at Duke.

Locating where a sound is coming from is partially learned with the aid of vision. Researchers sought to learn more about how the brain locates the source of a sound when the source is unclear and there are a number of possible visual matches.

"Our study is related to ventriloquism, in which the of a puppet's mouth 'captures' the sound of the puppeteer's voice," Groh said. "It is thought that one reason this illusion occurs is because vision normally teaches the brain how to tell where sounds are coming from. We investigated how the brain knows which visual stimulus should capture the location of a sound, such as why it is the puppet's mouth and not some other visual stimulus."

The study, which appears Thursday (Aug. 29) in the journal PLOS ONE, tested two competing . In one, the brain determines the location of a sound based on the simultaneous occurrence of audio and its visual source. In the other, the brain uses a "guess and check" method. In this scenario, visual feedback sent to the brain after the eye focuses on a sound affects how the eye searches for that sound in the future, possibly through the brain's reward-related .

In both paradigms, the visual stimulus—an LED—was displaced from the sound. Groh's team then looked for evidence that the LED caused a persistent mislocation of the sound.

"Surprisingly, we found that visual feedback exerts the more powerful effect on altering localization of sounds," Groh said. "This suggests that the active behavior of looking at the during a ventriloquism performance plays a role in causing the shift in where you hear the voice."

Participants in the study—11 humans and two rhesus monkeys—shifted their sight to a sound under different visual and audio scenarios.

In one scenario, called the "synchrony-only" task, a visual stimulus appeared at the same time as a sound but too briefly to provide feedback after an eye movement to that sound.

In another, the "feedback-only" task, the visual stimulus appeared during the execution of an eye movement to a sound, but was never on at the same time as the sound.

The study found that the "feedback-only task" exerted a much more powerful effect on the estimation of sound location, as measured with eye tracking, than did the other scenario. This suggests that those who have difficulty localizing sounds may benefit from practice involving eye movements.

On average, participants altered their eye movements in the direction of the lightsâ?? location to a greater degree, about a quarter of the way, when the was presented as feedback than when it was presented at the same time as the sound, the study found.

"This is about the brain's self-improvement skills," said co-author Daniel Pages, a graduate student in Psychology & Neuroscience at Duke. "What we're getting at is how the brain uses different types of information to improve how it does its job. In this case, it uses vision coupled with eye movements to improve hearing."

"We were surprised at how important the were," Groh said. "But finding sounds is really hard. Feedback about your performance is important for anything that is difficult, whether it is the B- you get on your homework or the error your eyes detect in localizing a ."

More information: Pages, D. and Groh, J. Looking at the Ventriloquist: Visual Outcome of eye Movements Calibrates Sound Localization, PLOS ONE, Aug. 29, 2013. DOI: 10.1371/journal.pone.0072562

Related Stories

Inner speech speaks volumes about the brain

Jul 16, 2013

Whether you're reading the paper or thinking through your schedule for the day, chances are that you're hearing yourself speak even if you're not saying words out loud. This internal speech—the monologue you "hear" inside ...

Recommended for you

Damage to brain networks affects stroke recovery

Nov 21, 2014

(Medical Xpress)—Initial results of an innovative study may significantly change how some patients are evaluated after a stroke, according to researchers at Washington University School of Medicine in St. ...

Dopamine leaves its mark in brain scans

Nov 21, 2014

Researchers use functional magnetic resonance imaging (fMRI) to identify which areas of the brain are active during specific tasks. The method reveals areas of the brain, in which energy use and hence oxygen ...

User comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.