How the brain retroactively makes sense of rapid auditory and visual sensory stimulation

October 9, 2018 by Lori Dajose, California Institute of Technology
Credit: California Institute of Technology

We experience the world through our senses, a constant torrent of sights, sounds, smells, and more. Our brains take these signals and process them, giving rise to our individual perceptions of the world. But sometimes our senses play tricks on us, notably in the case of perceptual illusions.

Now, Caltech researchers have developed two new illusions that reveal how the senses can influence each other—in particular, how sound can give rise to . These illusions occur so quickly that they illustrate a phenomenon called postdiction (as opposed to prediction) in which a stimulus that occurs later can retroactively affect our perceptions of an earlier event.

The Caltech work is among the first to show this kind of time-traveling across multiple senses.

The work was done in the laboratory of Shinsuke Shimojo, Gertrude Baltimore Professor of Experimental Psychology and affiliated faculty member of the Tianqiao and Chrissy Chen Institute for Neuroscience at Caltech. A paper describing the research appears in the October 3 issue of the journal PLOS ONE.

"Illusions are a really interesting window into the ," says first author Noelle Stiles (Ph.D. '15), a visitor in biology and biological engineering and a postdoctoral scholar–research associate at USC. "By investigating illusions, we can study the brain's decision-making process. For example, how does the brain determine reality with information from multiple senses that is at times noisy and conflicting? The brain uses assumptions about the environment to solve this problem. When these assumptions happen to be wrong, illusions can occur as the brain tries to make the best of a confusing situation. We can use these illusions to unveil the underlying inferences that the brain makes."

The two illusions in this study were developed to illustrate how stimuli that occur later can affect the perception of stimuli that have already occurred. Postdictive processing has been demonstrated within individual senses, but this work focuses on how the phenomenon can bridge multiple senses. The key to both of the new illusions is that the audio and visual stimuli occur rapidly, in under 200 milliseconds (one-fifth of a second). The brain, trying to make sense of this barrage of information, synthesizes the stimuli from both senses to determine the experience, using postdiction to do so.

Rabbit Illusion rc Watch and hear the Rabbit Illusion. Credit: California Institute of Technology

The first illusion is called the Illusory Rabbit. To produce the illusion, first a short beep and a quick flash are played nearly simultaneously on a computer, with the flash appearing at the left side of the screen. Next, 58 milliseconds after the first beep, a lone beep is played. Finally, 58 milliseconds after the second beep, a second nearly simultaneous beep-flash pair occurs, but with the flash appearing on the right side of the screen. The beep location is always central and does not move. Though only two flashes are played, most people viewing the illusion perceive three flashes, with an illusory flash coinciding with the second beep and appearing to be located in the center of the screen.

The fact that the illusory flash is perceived in between the left and right flashes is the key evidence that the brain is using postdictive processing.

"When the final beep-flash pair is later presented, the brain assumes that it must have missed the flash associated with the unpaired beep and quite literally makes up the fact that there must have been a second flash that it missed," explains Stiles. "This already implies a postdictive mechanism at work. But even more importantly, the only way that you could perceive the shifted illusory flash would be if the information that comes later in time—the final beep-flash combination—is being used to reconstruct the most likely location of the illusory flash as well."

The second illusion is called the Invisible Rabbit. In this related illusion, three flashes are shown on the screen, the first on the left, the second in the middle, and finally the third on the right, with only the first and third flashes coinciding with beeps. In this case, most people do not see the second flash—the one without a corresponding sound—at all. The absence of the second beep leads the brain to decide after the fact that there actually was no , even though it was in fact present.

By showing that a sound can excite a visual illusion, the researchers have uncovered new clues as to how the brain combines the senses over space and time to generate an integrated sense of perception.

"The significance of this study is twofold," says Shimojo. "First, it generalizes postdiction as a key process in perceptual processing for both a single sense and multiple senses. Postdiction may sound mysterious, but it is not—one must consider how long it takes the brain to process earlier , during which time subsequent stimuli from a different sense can affect or modulate the first. The second significance is that these illusions are among the very rare cases where sound affects vision, not vice versa, indicating dynamic aspects of neural processing that occur across space and time. These new illusions will enable researchers to identify optimal parameters for multisensory integration, which is necessary for both the design of ideal sensory aids and optimal training for low-vision individuals."

The paper is titled "What You Saw is What You Will Hear: Two New Illusions with Audiovisual Postdictive Effects."

Explore further: Illusion reveals that the brain fills in peripheral vision

More information: Noelle R. B. Stiles et al. What you saw is what you will hear: Two new illusions with audiovisual postdictive effects, PLOS ONE (2018). DOI: 10.1371/journal.pone.0204217

Related Stories

Illusion reveals that the brain fills in peripheral vision

December 8, 2016
What we see in the periphery, just outside the direct focus of the eye, may sometimes be a visual illusion, according to new findings published in Psychological Science, a journal of the Association for Psychological Science. ...

Imagining an object can change how we hear sounds later

April 12, 2018
Seeing an object at the same time that you hear sound coming from somewhere else can lead to the "ventriloquist illusion" and its aftereffect, but research suggests that simply imagining the object produces the same illusory ...

Brain appears to have different mechanisms for reconciling sight and sound

March 31, 2016
A new UCLA psychology study provides insights into how the brain combines sound and vision. The research suggests that there is not one sole mechanism in the brain that governs how much our senses work together to process ...

Animals could help reveal why humans fall for illusions

March 21, 2014
Visual illusions, such as the rabbit-duck and café wall are fascinating because they remind us of the discrepancy between perception and reality. But our knowledge of such illusions has been largely limited to studying humans.

Recommended for you

Dopamine's yin-yang personality: It's an upper and a downer

December 10, 2018
For decades, psychologists have viewed the neurotransmitter dopamine as a double-edged sword: released in the brain as a reward to train us to seek out pleasurable experiences, but also a "drug" the constant pursuit of which ...

Brainwaves suppress obvious ideas to help us think more creatively

December 10, 2018
The human brain needs to suppress obvious ideas in order to reach the most creative ones, according to scientists at Queen Mary University of London and Goldsmiths, University of London.

Regrowing damaged nerves hinges on shutting down key genes

December 10, 2018
Neurons in the brain and spinal cord don't grow back after injury, unlike those in the rest of the body. Cut your finger, and you'll probably be back to using it in days or weeks; slice through your spinal cord, and you likely ...

Editing consciousness: How bereaved people control their thoughts without knowing it

December 10, 2018
People who are grieving a major loss, such as the death of a spouse or a child, use different coping mechanisms to carry on with their lives. Psychologists have been able to track different approaches, which can reflect different ...

Regular problem solving does not protect against mental decline

December 10, 2018
The well known 'use it or lose it' claim has been widely accepted by healthcare professionals, but researchers in the Christmas issue of The BMJ find that regularly doing problem solving activities throughout your lifetime ...

Early career choices appear to influence personality, study finds

December 10, 2018
In the state of Baden-Wurttemberg, Germany, 16-year-old students in middle-track schools decide whether to stay in school to pursue an academic career or enroll in a vocational training program. A new study offers evidence ...

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.