Study aims to understand how, when the auditory system registers complex auditory-visual synchrony

October 23, 2013

Imagine the brain's delight when experiencing the sounds of Beethoven's "Moonlight Sonata" while simultaneously taking in a light show produced by a visualizer.

A new Northwestern University study did much more than that.

To understand how the responds to highly complex auditory-visual stimuli like music and moving images, the study tracked parts of the auditory system involved in the perceptual processing of "Moonlight Sonata" while it was synchronized with the light show made by the iTunes Jelly visualizer.

The study shows how and when the encodes auditory-visual synchrony between complex and changing sounds and images.

Much of related research looks at how the brain processes simple sounds and images. Locating a woodpecker in a tree, for example, is made easier when your brain combines the auditory (pecking) and visual (movement of the bird) streams and judges that they are synchronous. If they are, the brain decides that the two sensory inputs probably came from a single source.

While that research is important, Julia Mossbridge, lead author of the study and research associate in psychology at Northwestern, said it also is critical to expand investigations to highly complex stimuli like music and movies.

"These kinds of things are closer to what the brain actually has to manage to process in every moment of the day," she said. "Further, it's important to determine how and when sensory systems choose to combine stimuli across their boundaries.

"If someone's brain is mis-wired, sensory information could combine when it's not appropriate," she said. "For example, when that person is listening to a teacher talk while looking out a window at kids playing, and the auditory and visual streams are integrated instead of separated, this could result in confusion and misunderstanding about which go with what experience."

It was already known that the left auditory cortex is specialized to process sounds with precise, complex and rapid timing; this gift for auditory timing may be one reason that in most people, the left auditory cortex is used to process speech, for which timing is critical. The results of this study show that this specialization for timing applies not just to sounds, but to the timing of complex and dynamic sounds and images.

Previous research indicates that there are multi-sensory areas in the brain that link sounds and images when they change in similar ways, but much of this research is focused particularly on speech signals (e.g., lips moving as vowels and consonants are heard). Consequently, it hasn't been clear what areas of the brain process more general auditory-visual synchrony or how this processing differs when sounds and images should not be combined.

"It appears that the brain is exploiting the left 's gift at processing auditory timing, and is using similar mechanisms to encode auditory-visual synchrony, but only in certain situations; seemingly only when combining the sounds and images is appropriate," Mossbridge said.

Explore further: Rewired visual input to sound-processing part of the brain leads to compromised hearing

More information: The article "Seeing the song: Left auditory structures may track auditory-visual dynamic alignment" will appear Oct. 23 in PLOS ONE:

Related Stories

Getting an expected award music to the brain's ears

September 25, 2013

Several studies have shown that expecting a reward or punishment can affect brain activity in areas responsible for processing different senses, including sight or touch. For example, research shows that these brain regions ...

Multisensory integration: When correlation implies causation

December 15, 2011

In order to get a better picture of our surroundings, the brain has to integrate information from different senses, but how does it know which signals to combine? New research involving scientists from the Max Planck Institute ...

Recommended for you

Rat brain atlas provides MR images for stereotaxic surgery

October 21, 2016

Boris Odintsov, senior research scientist at the Biomedical Imaging Center at the Beckman Institute for Advanced Science and Technology at the University of Illinois in Urbana-Champaign, and Thomas Brozoski, research professor ...

ALS study reveals role of RNA-binding proteins

October 20, 2016

Although only 10 percent of amyotrophic lateral sclerosis (ALS) cases are hereditary, a significant number of them are caused by mutations that affect proteins that bind RNA, a type of genetic material. University of California ...

Imaging technique maps serotonin activity in living brains

October 20, 2016

Serotonin is a neurotransmitter that's partly responsible for feelings of happiness and for mood regulation in humans. This makes it a common target for antidepressants, which block serotonin from being reabsorbed by neurons ...

Overcoming egocentricity increases self-control

October 19, 2016

Neurobiological models of self-control usually focus on brain mechanisms involved in impulse control and emotion regulation. Recent research at the University of Zurich shows that the mechanism for overcoming egocentricity ...


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.