Why are sight and sound out of sync?

June 11, 2018, City University London

The way we process sight and sound are curiously out of sync by different amounts for different people and tasks, according to a new study from City, University of London.

When investigating the effect the researchers found that speech comprehension can sometimes actually improve by as much as 10 per cent when sound is delayed relative to vision, and that different individuals consistently have uniquely different optimal delays for different tasks.

As a result, the authors suggest that by tailoring sound delays on an individual basis via a or cochlear implant—or a setting on a computer media player—could have significant benefits for speech comprehension and enjoyment of multimedia. The study is published in Journal of Experimental Psychology: Human Perception and Performance.

When the researchers at City looked deeper into this phenomenon, they kept finding a very curious pattern: different tasks benefitted from opposite delays, even in the same person. For example, the more an individual's vision lags their audition in the performance of one task (e.g., identifying speech sounds), conversely the more their audition is likely to lag vision in other tasks (e.g., deciding whether lips followed or preceded the speaker's voice). This finding provides new insight into how we determine when events actually occur in the world and the nature of perceptual timing in the brain.

When we see and hear a person speak, travel via different pathways from our eyes and ears through the brain. The audiovisual asychronies measured in this study may occur because these sensory signals arrive at their different destinations in the brain at different times.

Yet how then do we ever know when the physical speech events actually happened in the world? The brain must have a way to solve this problem, given that we can still judge whether or not the original events are in sync with reasonable accuracy. For example, we are often able to easily identify when films have poor lip-sync.

Lead author Dr. Elliot Freeman, Senior Lecturer in the Department of Psychology at City, University of London, proposes a solution based on an analogous 'multiple clocks' problem: "Imagine standing in an antique shop full of clocks, and you want to know what the time is. Your best guess comes from the average across clocks. However, if one clock is particularly slow, others will seem fast relative to it.

"In our new theory, which we call 'temporal renormalisation', the 'clocks' are analogous to different mechanisms in the brain which each receive sight and sound out of sync: but if one such mechanism is subject to an auditory delay, this will bias the average, relative to which other mechanisms may seem to have a visual delay. This theory explains the curious finding that different tasks show opposite delays; it may also explain how we know when events in the world are actually happening, despite our brains having many conflicting estimates of their timing."

In their experiments, the researchers presented participants with audiovisual movies of a person speaking syllables, words or sentences, while varying the asynchrony of voice relative to lip movements. For each movie they measured their accuracy at identifying words spoken, or how strongly lip movements influenced what was heard.

In the latter case, the researchers exploited the McGurk illusion, where for example the phoneme 'ba' sounds like 'da' when mismatched with lip movements for 'ga'. They could then estimate the asynchrony that resulted in the maximal accuracy or strongest McGurk illusion. In a separate task, they also asked participants to judge whether the voice came before or after the lip movements, from which they could estimate the subjective asynchrony.

Speaking about the study, Dr. Freeman said: "We often assume that the best way to comprehend speech is to match up what we hear with lip movements, and that this works best when sight and sound are simultaneous. However, our new study confirms that sight and sound really are out of sync by different amounts in different people. We also found that for some individuals, manually delaying voices relative to could improve speech comprehension and the accuracy of word identification by 10% or more.

"This paper also introduces a new automated method for assessing individual audiovisual asynchronies, which could be administered over the internet or via an 'app'. Once an individual's perceptual asynchrony is measured, it may be corrected artificially with a tailored delay. This could be implemented via a hearing aid or cochlear implant, or a setting on a computer media player, with potential benefits for and enjoyment of multimedia.

"Asynchronous perception may impact on cognitive performance, and future studies could examine its associations with schizotypal personality traits, autism spectrum traits, and dyslexia."

Explore further: New research suggests speech comprehension improves with a slight auditory delay in some individuals

Related Stories

New research suggests speech comprehension improves with a slight auditory delay in some individuals

April 21, 2017
A new paper has shed light on sensory timing in the brain by showing that there is a lag between us hearing a person's voice and seeing their lips move, and these delays vary depending on what task we're doing.

Learning a second language alters sensory perception, study finds

May 14, 2018
Learning a second language can change the way our senses work together to interpret speech, according to a new Northwestern University study.

The man who hears you speak before he sees your lips move

July 8, 2013
Research led by Dr Elliot Freeman, from City University London's Department of Psychology, which examined the first documented case of someone who hears people talk before he sees their lips move, has been published in New ...

When your eyes override your ears: New insights into the McGurk effect

February 16, 2017
Seeing is not always believing - visual speech (mouth movements) mismatched with auditory speech (sounds) can result in the perception of an entirely different message. This mysterious illusion is known as the McGurk effect. ...

Visual cues amplify sound

February 13, 2018
Looking at someone's lips is good for listening in noisy environments because it helps our brains amplify the sounds we're hearing in time with what we're seeing, finds a new UCL-led study.

Recommended for you

Cell type and environment influence protein turnover in the brain

June 19, 2018
Scientists have revealed that protein molecules in the brain are broken down and replaced at different rates, depending on where in the brain they are.

Researchers investigate changes in white matter in mice exposed to low-frequency brain stimulation

June 19, 2018
A team of researchers at the University of Oregon has learned more about the mechanism involved in mouse brain white matter changes as it responds to stimulation. In their paper published in Proceedings of the National Academy ...

Left, right and center: mapping emotion in the brain

June 19, 2018
According to a radical new model of emotion in the brain, a current treatment for the most common mental health problems could be ineffective or even detrimental to about 50 percent of the population.

Often overlooked glial cell is key to learning and memory

June 18, 2018
Glial cells surround neurons and provide support—not unlike hospital staff and nurses supporting doctors to keep operations running smoothly. These often-overlooked cells, which include oligodendrocytes and astrocytes, ...

Neuroscientists map brain's response to cold touch

June 18, 2018
Carnegie Mellon University neuroscientists have mapped the feeling of cool touch to the brain's insula in a mouse model. The findings, published in the June 15 issue of Journal of Comparative Neurology, provide an experimental ...

Electrically stimulating the brain may restore movement after stroke

June 18, 2018
UC San Francisco scientists have improved mobility in rats that had experienced debilitating strokes by using electrical stimulation to restore a distinctive pattern of brain cell activity associated with efficient movement. ...

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.