Multisensory integration: When correlation implies causation

This is the experimental setup. Credit: Kopp/ Parise

In order to get a better picture of our surroundings, the brain has to integrate information from different senses, but how does it know which signals to combine? New research involving scientists from the Max Planck Institute for Biological Cybernetics, the Bernstein Center for Computational Neuroscience Tübingen, the University of Oxford, and the University of Bielefeld has demonstrated that humans exploit the correlation between the temporal structures of signals to decide which of them to combine and which to keep segregated. This research is about to be published in Current Biology.

Multisensory signals originating from the same distal event are often similar in nature. Think of fireworks on New Year's Eve, an object falling and bouncing on the floor, or the footsteps of a person walking down the street. The temporal structures of such visual and auditory events are always almost overlapping (i.e., they correlate), and we often effortlessly assume an underlying unity between our visual and auditory experiences. In fact, the similarity of temporal structure of multiple unisensory signals, rather than merely their temporal coincidence as it has been previously thought, provides a potentially powerful cue for the to determine whether or not multiple have a common cause.

Cesare Parise from the Max Planck Institute for in Tübingen and Bernstein Center for Computational Neuroscience Tübingen and his colleagues set out to examine the role of signal correlation in multisensory integration by asking people to localize a stream of beeps and flashes. Participants seated in front of a large screen where sounds (streams of noise bursts) and images (streams blurred blobs) were presented from different spatial locations. On some trials only visual or were presented, while on other trials visual and auditory stimuli were presented in combination. Critically, on combined audiovisual trials, the temporal structure of the visual and auditory stimuli could either be correlated or not. Participants were required to report the spatial position of such stimuli by moving a cursor controlled by a graphic tablet. In line with previous studies, participants were more precise when the auditory and visual streams were presented together than when they were presented in isolation. Notably, precision was even higher when auditory and visual streams were correlated, and closely approached the theoretical maximum.

These results demonstrate that humans optimally combine multiple sensory signals only when they correlate in time. Previous research has demonstrated that optimal integration only occurs when the brain is sure that the signals have a common underlying cause. These results therefore demonstrate that the brain uses the statistical correlation between the sensory signals to infer whether they have a common physical cause, and hence whether they provide redundant information that should be integrated.

The researchers suggest the brain has evolved this ability to combine potentially related information from different senses so it can effectively pick its way through the noisy environments of everyday life.

"It's why at a noisy cocktail party you can tell who is speaking with which voice," says Parise. "Our eyes and ears are continually taking in sensory information and our brains make of it all by merging together sights and sounds with similar temporal structures."

In spite of being a pervasive aspect of sensory processing, little is known about the low-level statistical determinants of multisensory integration for signals with complex dynamic temporal patterns. This research highlights the role of a key organizational principle for multisensory perceptual grouping. What at first glance appears to be a logical fallacy, namely inferring causation from correlation, turns out to be the rule in perception.

More information: Cesare V. Parise, Charles Spence, Marc O. Ernst: When Correlation Implies Causation in Multisensory Integration. Current Biology doi: 10.1016/j.cub.2011.11.039

Related Stories

Where the brain combines what's heard and felt

Oct 19, 2005

Using functional magnetic resonance imaging, researchers at the Max Planck Institute for Biological Cybernetics in Tübingen, Germany have showed that the integration of auditory and touch information takes place in the 'hearing ...

Wired for sound: How the brain senses visual illusions

Apr 11, 2007

In a study that could help reveal how illusions are produced in the brain's visual cortex, researchers at the UCSD School of Medicine have found new evidence of rapid integration of auditory and visual sensations in the brain. ...

Do you hear what i see?

Feb 20, 2007

New research pinpoints specific areas in sound processing centers in the brains of macaque monkeys that shows enhanced activity when the animals watch a video.

How our senses combine to give us a better view of the world

Nov 12, 2008

From a young age we are taught about the five senses and how they help us to explore our world. Although each sense seems to be its own entity, recent studies have indicated that there is actually a lot of overlap and blending ...

Recommended for you

New test to help brain injury victims recover

22 hours ago

A dynamic new assessment for helping victims of trauma to the brain, including those suffering from progressive conditions such as dementia, has been developed by a clinical neuropsychologist at the University ...

See-through sensors open new window into the brain

Oct 21, 2014

(Medical Xpress)—Developing invisible implantable medical sensor arrays, a team of University of Wisconsin-Madison engineers has overcome a major technological hurdle in researchers' efforts to understand ...

User comments