Study aims to understand how, when the auditory system registers complex auditory-visual synchrony

October 23, 2013

Imagine the brain's delight when experiencing the sounds of Beethoven's "Moonlight Sonata" while simultaneously taking in a light show produced by a visualizer.

A new Northwestern University study did much more than that.

To understand how the responds to highly complex auditory-visual stimuli like music and moving images, the study tracked parts of the auditory system involved in the perceptual processing of "Moonlight Sonata" while it was synchronized with the light show made by the iTunes Jelly visualizer.

The study shows how and when the encodes auditory-visual synchrony between complex and changing sounds and images.

Much of related research looks at how the brain processes simple sounds and images. Locating a woodpecker in a tree, for example, is made easier when your brain combines the auditory (pecking) and visual (movement of the bird) streams and judges that they are synchronous. If they are, the brain decides that the two sensory inputs probably came from a single source.

While that research is important, Julia Mossbridge, lead author of the study and research associate in psychology at Northwestern, said it also is critical to expand investigations to highly complex stimuli like music and movies.

"These kinds of things are closer to what the brain actually has to manage to process in every moment of the day," she said. "Further, it's important to determine how and when sensory systems choose to combine stimuli across their boundaries.

"If someone's brain is mis-wired, sensory information could combine when it's not appropriate," she said. "For example, when that person is listening to a teacher talk while looking out a window at kids playing, and the auditory and visual streams are integrated instead of separated, this could result in confusion and misunderstanding about which go with what experience."

It was already known that the left auditory cortex is specialized to process sounds with precise, complex and rapid timing; this gift for auditory timing may be one reason that in most people, the left auditory cortex is used to process speech, for which timing is critical. The results of this study show that this specialization for timing applies not just to sounds, but to the timing of complex and dynamic sounds and images.

Previous research indicates that there are multi-sensory areas in the brain that link sounds and images when they change in similar ways, but much of this research is focused particularly on speech signals (e.g., lips moving as vowels and consonants are heard). Consequently, it hasn't been clear what areas of the brain process more general auditory-visual synchrony or how this processing differs when sounds and images should not be combined.

"It appears that the brain is exploiting the left 's gift at processing auditory timing, and is using similar mechanisms to encode auditory-visual synchrony, but only in certain situations; seemingly only when combining the sounds and images is appropriate," Mossbridge said.

Explore further: Rewired visual input to sound-processing part of the brain leads to compromised hearing

More information: The article "Seeing the song: Left auditory structures may track auditory-visual dynamic alignment" will appear Oct. 23 in PLOS ONE: dx.plos.org/10.1371/journal.pone.0077201

Related Stories

Rewired visual input to sound-processing part of the brain leads to compromised hearing

August 22, 2012
Scientists at Georgia State University have found that the ability to hear is lessened when, as a result of injury, a region of the brain responsible for processing sounds receives both visual and auditory inputs.

The importance of keeping a beat: Researchers link ability to keep a beat to reading, language skills

September 17, 2013
The findings of a Northwestern University study of more than 100 high school students lend proof to the surprising link between music, rhythmic abilities and language skills.

Getting an expected award music to the brain's ears

September 25, 2013
Several studies have shown that expecting a reward or punishment can affect brain activity in areas responsible for processing different senses, including sight or touch. For example, research shows that these brain regions ...

Brain wiring quiets the voice inside your head

September 3, 2013
During a normal conversation, your brain is constantly adjusting the volume to soften the sound of your own voice and boost the voices of others in the room.

Multisensory integration: When correlation implies causation

December 15, 2011
In order to get a better picture of our surroundings, the brain has to integrate information from different senses, but how does it know which signals to combine? New research involving scientists from the Max Planck Institute ...

Unraveling the mysteries of the maternal brain: Odors influence the response to sounds

October 19, 2011
Motherhood is associated with the acquisition of a host of new behaviors that must be driven, at least in part, by alterations in brain function. Now, new research published by Cell Press in the October 20 issue of the journal ...

Recommended for you

Running on autopilot: Scientists find important new role for 'daydreaming' network

October 23, 2017
A brain network previously associated with daydreaming has been found to play an important role in allowing us to perform tasks on autopilot. Scientists at the University of Cambridge showed that far from being just 'background ...

Rhythm of memory: Inhibited neurons set the tempo for memory processes

October 23, 2017
The more we know about the billions of nerve cells in the brain, the less their interaction appears spontaneous and random. The harmony underlying the processing of memory contents has been revealed by Prof. Dr. Marlene Bartos' ...

Researchers demonstrate 'mind-reading' brain-decoding tech

October 23, 2017
Researchers have demonstrated how to decode what the human brain is seeing by using artificial intelligence to interpret fMRI scans from people watching videos, representing a sort of mind-reading technology.

Research revises our knowledge of how the brain learns to fear

October 23, 2017
Our brains wire themselves up during development according to a series of remarkable genetic programs that have evolved over millions of years. But so much of our behavior is the product of things we learn only after we emerge ...

Scientists use supercomputer to search for "memory molecules"

October 23, 2017
Until now, searching for genes related to memory capacity has been comparable to seeking out the proverbial "needle in a haystack." Scientists at the University of Basel made use of the CSCS supercomputer Piz Daint to discover ...

High-speed locomotion neurons found in the brainstem

October 23, 2017
Think of taking a casual stroll on a sunny Sunday afternoon or running at full speed to catch a bus for work on Monday morning as two extremes. Both forms of locomotion entail a perfect interplay between arms and legs, yet ...

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.