The neural underpinnings of surround sound and its effects on a listener's sense of presence

Over the past few decades, engineers have introduced and perfected numerous technologies designed to produce highly immersive experiences, including virtual reality (VR), augmented reality (AR) and advanced surround sound systems. While these technologies are now widely used worldwide, their effects on human perceptions and their associated brain processes have rarely been explored scientifically.

Researchers at University of Parma, State University of Milan and Sapienza University of Rome recently carried out a study aimed at better understanding the neural correlates of surround sound and their impact on a human listener's sense of embodiment and presence. Their paper, published in Frontiers in Neuroscience, is part of a set of works into cinematic aesthetics applied to neuroscience (i.e., neuroaesthetics) conducted by Prof. Vittorio Gallese's lab at University of Parma.

"My personal interest is in acoustic perception, and the role of sound in the cinematic experience has been largely overlooked," Nunzio Langiulli, one of the researchers who carried out the study, told Medical Xpress.

"Even if empirical research on the relationship between moving images and sounds in cinema is limited, some authors have suggested that sound could enhance the immersive qualities of cinematic experience enhancing the sense of presence. So, we designed a study aimed to investigate the time course and neural correlates of the sense of presence as evoked by different audio presentation modes during cinematic immersion."

To explore the impact of surround sound on both perception and the brain, Langiulli, Prof. Gallese and their colleagues carried out a series of experiments involving human participants. Their study participants were asked to sit and listen to different extracts from movies, which were presented to them in different ways (i.e., through a monophonic, stereo, or surround system).

"We first designed a behavioral experiment and subsequently a high-density electroencephalographic (HD- EEG) experiment," Langiulli explained. "In the first experiment, we better characterized the sense of presence related to acoustic spatialized presentation in terms of enjoyment, emotional involvement, physical immersion, and realism. In the second experiment, we adopted a brain imaging technique to investigate the neural correlate of the sense of presence."

In their first experiment, participants were asked to share feedback about their experience of the audio recordings they had listened to, answering questions about how much they enjoyed the experience, how engaged they felt, and so on. In the second experiment, the researchers recorded the participants' brain activity using electroencephalography (EEG), a technique that measures electrical impulses in the brain.

Through these two experiments, Langiulli and his colleagues gathered interesting new insight about the relationship between virtual acoustic spatial environments and a human listener's sense of presence. Most notably, they showed that when participants listened to recordings through a surround sound system embodied simulation mechanisms in their brain were more active. These neural mechanisms could be viewed as physiological correlates of the sense of presence they experienced.

"We attained a deeper comprehension of how our brain processes acoustic immersive cues. Further findings on how we integrate spatialized sound and visual information could also be applied to other areas that rely on the integration of sound and visual information to create immersive experiences, such as virtual reality and augmented reality," Langiulli said.

"Filmmakers and sound designers may also leverage this knowledge to precisely manipulate audio-visual elements, resulting in a heightened emotional impact and greater engagement with film scenes. The knowledge gained from this exploration should also have broader implications in fields beyond entertainment."

These recent findings gathered by Langiulli, Prof. Gallese and their collaborators could potentially inform the development of new entertainment content and technologies designed to boost a user's sense of presence. In addition, their work could have implications for the fields of education and medicine, for instance guiding the design of effective educational multimedia content and immersive training for therapeutic or rehabilitation purposes.

"In the future we aim to expand our comprehension of the spatialized acoustic perception using a different set of stimuli," Langiulli added. "Simpler acoustic cues can allow us to fully understand the underlying mechanisms and factors that contribute to the neural response that we observed and analyzed in this study."

© 2023 Science X Network