Sound and vision work hand in hand, psychologists report

December 8, 2011, University of California - Los Angeles

Our senses of sight and hearing work closely together, perhaps more than people realize, a new UCLA psychology study shows.

"If we think of the perceptual system as a where each sense is like a person casting a and all votes are counted to reach a decision — although not all votes are counted equally — what our study shows is that the voters talk to one another and influence one another even before each casts a vote," said Ladan Shams, a UCLA associate professor of psychology and the senior author of the new study.

In the study of how one sense can affect another, Shams, an expert on perception and cognitive neuroscience, and her colleagues showed 63 participants a large number of dots on a screen in two separate phases, with a break between the phases. In one phase, the dots moved around randomly; in the other, some of the dots moved together from right to left. In both phases, the dots were accompanied by sound.

Over a series of experiments, the researchers asked the participants to correctly identify the phase in which the dots moved together horizontally.

Participants were divided into three groups. One group heard sound moving from right to left as the dots moved from right to left, and they heard sound that remained stationary during the random phase. A second group heard the right-to-left sound during both phases. And a third group heard sound moving in the opposite direction — from left to right — during both phases. Then, each participant experienced trials in all three conditions.

As Shams expected, the participants were best able to identify the phase in which the dots moved horizontally when the sound moved in the same direction as the dots but remained stationary during the random phase. The researchers found that the sound that moved in the opposite direction neither enhanced nor worsened the participants' visual perception.

Surprisingly, the sound that traveled leftward both when the dots moved leftward and when the dots moved randomly — that is, sound that provided no useful information for choosing between the two phases — also helped people correctly choose the phase with the horizontal motion. Because the sound was identical in both phases, if the participants closed their eyes they, would have a 50-50 chance of successfully performing the task based on sound alone; with their eyes open, however, the interaction between sound and vision led to a significant improvement in detection of visual motion. Hearing enhanced seeing.

"Imagine you are playing ping-pong with a friend who serves the ball," Shams said. "You receive information about where and when the ball hit the table by both vision and hearing. Scientists have believed that each of the senses produces an estimate relevant for the task, and then these votes get combined subconsciously according to rules that take into account which sense is more reliable. This is how the senses interact in how we perceive the world. However, our findings show that the senses of hearing and vision can also interact at a more basic level, before they each even produce an estimate."

The study appears in the December issue of Psychological Science, a journal published by the Association for Psychological Science.

The senses affect one another other in many ways, said Robyn Kim, a former UCLA psychology postdoctoral scholar in Shams' laboratory and lead author of the study. There are connections between the auditory and visual portions of the brain at the cognitive level. When the information from one sense is ambiguous, another can step in and clarify or ratify the perception, Shams and Kim said.

"Most of us understand that smell affects taste," said Kim, who added that sound similarly affects vision.

"This study shows that at least in regards to perception of moving objects, hearing and sight are deeply intertwined, to the degree that even when is completely irrelevant to the task, it still influences the way we see the world," Shams said.

UCLA psychology graduate student Megan Peters, who conducts research in Shams' laboratory, was a co-author on the study.

Explore further: Even unconsciously, sound helps us see

Related Stories

Even unconsciously, sound helps us see

December 2, 2011
“Imagine you are playing ping-pong with a friend. Your friend makes a serve. Information about where and when the ball hit the table is provided by both vision and hearing. Scientists have believed that each of the ...

New evidence of an unrecognized visual process

November 24, 2011
(Medical Xpress) -- We don’t see only what meets the eye. The visual system constantly takes in ambiguous stimuli, weighs its options, and decides what it perceives. This normally happens effortlessly. Sometimes, however, ...

Up, down, right, left -- how visual cues help us understand bodily motion

July 6, 2011
(Medical Xpress) -- "Our visual system is tuned towards perceiving other people. We spend so much time doing that—seeing who they are, what they are doing, what they intend to do," says psychology professor Nikolaus ...

'Feeling' sound: The sense of hearing and touch may have evolved together

June 2, 2011
Lying in bed at night, one of the worst sounds a person can hear is the buzz of a nearby mosquito. Concentrating on the buzzing might keep you from falling asleep, but it also seems to heighten the awareness of your skin ...

Memories may skew visual perception

July 20, 2011
Taking a trip down memory lane while you are driving could land you in a roadside ditch, new research indicates. Vanderbilt University psychologists have found that our visual perception can be contaminated by memories of ...

Recommended for you

Some gut feelings are a red flag, according to research

March 21, 2018
Are you guided by gut feelings?

20-year study finds little change in social functioning in people with psychosis

March 21, 2018
Researchers have found that levels of social impairment in people with schizophrenia remained remarkably stable in the years after the first hospitalisation for psychosis.

Treating depression in cancer increases quality of life, but not length of life

March 21, 2018
Researchers have found that treating depression doesn't make cancer patients live longer, but it does make lives immeasurably better.

We start caring about our reputations as early as kindergarten

March 20, 2018
Kindergarteners don't use social media, but they do care about their public image. Research suggests that by the time kids go to elementary school, they're thinking critically about their reputation. In a Review published ...

We can read each other's emotions from surprisingly tiny changes in facial color, study finds

March 19, 2018
Our faces broadcast our feelings in living color—even when we don't move a muscle.

Social media use at age 10 could reduce wellbeing of adolescent girls

March 19, 2018
Social media use may have different effects on wellbeing in adolescent boys and girls, according to research published in the open access journal BMC Public Health.


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.