Sensory substitution illuminates cognitive processes while offering therapeutic applications

June 11, 2018, CORDIS
Sensory substitution illuminates cognitive processes while offering therapeutic applications
Credit: Monstar Studio, Shutterstock

What are the implications of being able to "listen to a choreography" or "feel a ballet?" Despite the importance of observing the bodily movements of others to understand and predict actions, little is known about the underlying plasticity of the neural mechanism.

The blending of the arts and sciences, traditionally viewed as being somewhat at cross-purposes, has found fertile ground in areas such as education, therapy and rehabilitation. The EU-funded DANCE project found this to be especially true when combined with human-centred computing innovations.

The DANCE project explored how bodily movement can be expressed and perceived in auditory terms. The project developed technologies which measured the expressive qualities of full-body movement in real-time, translating gestures from visual to auditory phenomena. While contributing to brain-imaging which maps the visual and auditory cortex of blind and sighted people, the approach also offers therapeutic potential, as well as novel human-machine non-verbal interfaces.

Seeing through listening

Research has shown that the sonification (using non-speech audio to convey information) of movement can increase body awareness, along with heightened sensitivity to emotions. This finding has proven useful when applied to therapeutic endeavours.

DANCE brought together performance artists, sound designers, vision and social neuroscientists, computer scientists and engineers to further explore the potential for rehabilitation and increased social inclusion emerging from the sharing of spaces and emotions between visually impaired and sighted people. The EU-funded DANCE team have previously adopted the approach to help patients with Parkinson's disease and those suffering from chronic pain.

In the project's work, choreographed expressive movement actually creates sonic content in real time. As project coordinator Dr. Antonio Camurri, explains, "This means that itself is then conceived as a musical composition, changing its traditional dimension to offer a listening experience rather than principally a visual one."

An overarching theme of the project was to explore the way in which multisensory information runs the risks of creating sensory overload in perceivers. "So instead of augmenting the senses, with the consequent risk of blunting the experience, DANCE proceeded by deliberately creating sensory deprivation as a means of sharpening the senses." Dr. Camurri elaborates. In doing so DANCE also contributed to neuroscience studies looking at brain plasticity.

DANCE conducted a human imaging (fMRI) study investigating the relationship between movement features, obtained from automated computational analyses of video clips, and the brain activity during perception of the videos.

The results showed that features requiring low-level computation, such as movement speed, map to specific brain areas related to early visual and motion sensitive regions. While mid-level features, such as expressive fluidity, are related to dynamic aspects of posture encoded in different brain areas.

As Dr. Camurri summarises, "Our computational feature based analysis suggests that the neural mechanism of movement encoding uses features that have a different neural basis from semantic categories. That is, movement perception is organized in the brain not so much by semantic categories, than by features of the body movements themselves."

Choreographing the next steps

The team developed a technology platform alongside freely available software libraries. These have been adopted for several public events, scientific experiments and mobile applications. A publicly available movement dataset was also created along with prototypes of applications for both blind and non-blind people.

The project's main public event entitled, "Atlante del gesto Genova' included the participation of blind and sighted people, with an institute for blind people, which took place from November 2016 to March 2017 and has created an on-going community of over 150 participants (including non-specialist members of the public).

The work can be adapted to generate creative content for therapy and rehabilitation. For example, a new joint laboratory with the Gaslini children's hospital in Italy was created during DANCE. Here, physiotherapists alongside computer engineers and researchers, collaborated in the creation of gaming for rehabilitation.

DANCE has also been a rich source of inspiration for further research directions such as neuroscientific studies seeking to reveal the underlying involved in the perception of high-level, affective communication through , with the aim of creating generation of novel interfaces and applications.

Explore further: Dan­cer's brains dis­play brain fre­quen­cies linked to emo­tion and memory pro­cesses

Related Stories

Dan­cer's brains dis­play brain fre­quen­cies linked to emo­tion and memory pro­cesses

May 4, 2018
Neuroscience has studied music for decades, and it has been found to activate both the cortical and deeper brain areas. Neuroscience of dance, instead, is a young but quickly growing field.

Dance aids healthier ageing

April 5, 2018
Queensland Ballet and QUT today released the results of a joint project examining the health and wellbeing benefits of ballet for older Australians.

Research suggests that the arts might play a role in empathy training

November 17, 2016
A recent study from researchers at City, University of London and the University of the Balearic Islands suggests that dancers are more emotionally sensitive than the rest of us.

How honeybees read the waggle dance

October 9, 2017
Neurons that enable honeybees to sense the waggle dance—a form of symbolic communication used by female bees to inform the hivemates about the location of a food source—are investigated in new research published in Journal ...

Brains of frequent dance spectators exhibit motor mirroring while watching familiar dance

March 21, 2012
Experienced ballet spectators with no physical expertise in ballet showed enhanced muscle-specific motor responses when watching live ballet, according to a Mar. 21 report in the open access journal PLoS ONE.

Recommended for you

Cell type and environment influence protein turnover in the brain

June 19, 2018
Scientists have revealed that protein molecules in the brain are broken down and replaced at different rates, depending on where in the brain they are.

Researchers investigate changes in white matter in mice exposed to low-frequency brain stimulation

June 19, 2018
A team of researchers at the University of Oregon has learned more about the mechanism involved in mouse brain white matter changes as it responds to stimulation. In their paper published in Proceedings of the National Academy ...

Left, right and center: mapping emotion in the brain

June 19, 2018
According to a radical new model of emotion in the brain, a current treatment for the most common mental health problems could be ineffective or even detrimental to about 50 percent of the population.

Often overlooked glial cell is key to learning and memory

June 18, 2018
Glial cells surround neurons and provide support—not unlike hospital staff and nurses supporting doctors to keep operations running smoothly. These often-overlooked cells, which include oligodendrocytes and astrocytes, ...

Neuroscientists map brain's response to cold touch

June 18, 2018
Carnegie Mellon University neuroscientists have mapped the feeling of cool touch to the brain's insula in a mouse model. The findings, published in the June 15 issue of Journal of Comparative Neurology, provide an experimental ...

Electrically stimulating the brain may restore movement after stroke

June 18, 2018
UC San Francisco scientists have improved mobility in rats that had experienced debilitating strokes by using electrical stimulation to restore a distinctive pattern of brain cell activity associated with efficient movement. ...


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.