Sound and vision work hand in hand, psychologists report

December 8, 2011

Our senses of sight and hearing work closely together, perhaps more than people realize, a new UCLA psychology study shows.

"If we think of the perceptual system as a where each sense is like a person casting a and all votes are counted to reach a decision — although not all votes are counted equally — what our study shows is that the voters talk to one another and influence one another even before each casts a vote," said Ladan Shams, a UCLA associate professor of psychology and the senior author of the new study.

In the study of how one sense can affect another, Shams, an expert on perception and cognitive neuroscience, and her colleagues showed 63 participants a large number of dots on a screen in two separate phases, with a break between the phases. In one phase, the dots moved around randomly; in the other, some of the dots moved together from right to left. In both phases, the dots were accompanied by sound.

Over a series of experiments, the researchers asked the participants to correctly identify the phase in which the dots moved together horizontally.

Participants were divided into three groups. One group heard sound moving from right to left as the dots moved from right to left, and they heard sound that remained stationary during the random phase. A second group heard the right-to-left sound during both phases. And a third group heard sound moving in the opposite direction — from left to right — during both phases. Then, each participant experienced trials in all three conditions.

As Shams expected, the participants were best able to identify the phase in which the dots moved horizontally when the sound moved in the same direction as the dots but remained stationary during the random phase. The researchers found that the sound that moved in the opposite direction neither enhanced nor worsened the participants' visual perception.

Surprisingly, the sound that traveled leftward both when the dots moved leftward and when the dots moved randomly — that is, sound that provided no useful information for choosing between the two phases — also helped people correctly choose the phase with the horizontal motion. Because the sound was identical in both phases, if the participants closed their eyes they, would have a 50-50 chance of successfully performing the task based on sound alone; with their eyes open, however, the interaction between sound and vision led to a significant improvement in detection of visual motion. Hearing enhanced seeing.

"Imagine you are playing ping-pong with a friend who serves the ball," Shams said. "You receive information about where and when the ball hit the table by both vision and hearing. Scientists have believed that each of the senses produces an estimate relevant for the task, and then these votes get combined subconsciously according to rules that take into account which sense is more reliable. This is how the senses interact in how we perceive the world. However, our findings show that the senses of hearing and vision can also interact at a more basic level, before they each even produce an estimate."

The study appears in the December issue of Psychological Science, a journal published by the Association for Psychological Science.

The senses affect one another other in many ways, said Robyn Kim, a former UCLA psychology postdoctoral scholar in Shams' laboratory and lead author of the study. There are connections between the auditory and visual portions of the brain at the cognitive level. When the information from one sense is ambiguous, another can step in and clarify or ratify the perception, Shams and Kim said.

"Most of us understand that smell affects taste," said Kim, who added that sound similarly affects vision.

"This study shows that at least in regards to perception of moving objects, hearing and sight are deeply intertwined, to the degree that even when is completely irrelevant to the task, it still influences the way we see the world," Shams said.

UCLA psychology graduate student Megan Peters, who conducts research in Shams' laboratory, was a co-author on the study.

Explore further: Even unconsciously, sound helps us see

Related Stories

Even unconsciously, sound helps us see

December 2, 2011
“Imagine you are playing ping-pong with a friend. Your friend makes a serve. Information about where and when the ball hit the table is provided by both vision and hearing. Scientists have believed that each of the ...

New evidence of an unrecognized visual process

November 24, 2011
(Medical Xpress) -- We don’t see only what meets the eye. The visual system constantly takes in ambiguous stimuli, weighs its options, and decides what it perceives. This normally happens effortlessly. Sometimes, however, ...

Up, down, right, left -- how visual cues help us understand bodily motion

July 6, 2011
(Medical Xpress) -- "Our visual system is tuned towards perceiving other people. We spend so much time doing that—seeing who they are, what they are doing, what they intend to do," says psychology professor Nikolaus ...

'Feeling' sound: The sense of hearing and touch may have evolved together

June 2, 2011
Lying in bed at night, one of the worst sounds a person can hear is the buzz of a nearby mosquito. Concentrating on the buzzing might keep you from falling asleep, but it also seems to heighten the awareness of your skin ...

Memories may skew visual perception

July 20, 2011
Taking a trip down memory lane while you are driving could land you in a roadside ditch, new research indicates. Vanderbilt University psychologists have found that our visual perception can be contaminated by memories of ...

Recommended for you

Gene associated with schizophrenia risk regulates neurodevelopment

September 25, 2017
A gene associated with the risk of schizophrenia regulates critical components of early brain development, according to a new study led by researchers from Penn State University. The gene is involved in the translation of ...

Child abuse affects brain wiring

September 25, 2017
Researchers from the McGill Group for Suicide Studies, based at the Douglas Mental Health University Institute and McGill University's Department of Psychiatry, have just published research in the American Journal of Psychiatry ...

For a better 'I,' there needs to be a supportive 'we'

September 25, 2017
If you're one of those lucky individuals with high motivation and who actively pursues personal growth goals, thank your family and friends who support you.

Babies can learn that hard work pays off

September 21, 2017
If at first you don't succeed, try, try again. A new study from MIT reveals that babies as young as 15 months can learn to follow this advice. The researchers found that babies who watched an adult struggle at two different ...

Study links brain inflammation to suicidal thinking in depression

September 21, 2017
Patients with major depressive disorder (MDD) have increased brain levels of a marker of microglial activation, a sign of inflammation, according to a new study in Biological Psychiatry by researchers at the University of ...

Oxytocin turns up the volume of your social environment

September 20, 2017
Before you shop for the "cuddle" hormone oxytocin to relieve stress and enhance your social life, read this: a new study from the University of California, Davis, suggests that sometimes, blocking the action of oxytocin in ...

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.