Sensory substitution illuminates cognitive processes while offering therapeutic applications

June 11, 2018, CORDIS
Sensory substitution illuminates cognitive processes while offering therapeutic applications
Credit: Monstar Studio, Shutterstock

What are the implications of being able to "listen to a choreography" or "feel a ballet?" Despite the importance of observing the bodily movements of others to understand and predict actions, little is known about the underlying plasticity of the neural mechanism.

The blending of the arts and sciences, traditionally viewed as being somewhat at cross-purposes, has found fertile ground in areas such as education, therapy and rehabilitation. The EU-funded DANCE project found this to be especially true when combined with human-centred computing innovations.

The DANCE project explored how bodily movement can be expressed and perceived in auditory terms. The project developed technologies which measured the expressive qualities of full-body movement in real-time, translating gestures from visual to auditory phenomena. While contributing to brain-imaging which maps the visual and auditory cortex of blind and sighted people, the approach also offers therapeutic potential, as well as novel human-machine non-verbal interfaces.

Seeing through listening

Research has shown that the sonification (using non-speech audio to convey information) of movement can increase body awareness, along with heightened sensitivity to emotions. This finding has proven useful when applied to therapeutic endeavours.

DANCE brought together performance artists, sound designers, vision and social neuroscientists, computer scientists and engineers to further explore the potential for rehabilitation and increased social inclusion emerging from the sharing of spaces and emotions between visually impaired and sighted people. The EU-funded DANCE team have previously adopted the approach to help patients with Parkinson's disease and those suffering from chronic pain.

In the project's work, choreographed expressive movement actually creates sonic content in real time. As project coordinator Dr. Antonio Camurri, explains, "This means that itself is then conceived as a musical composition, changing its traditional dimension to offer a listening experience rather than principally a visual one."

An overarching theme of the project was to explore the way in which multisensory information runs the risks of creating sensory overload in perceivers. "So instead of augmenting the senses, with the consequent risk of blunting the experience, DANCE proceeded by deliberately creating sensory deprivation as a means of sharpening the senses." Dr. Camurri elaborates. In doing so DANCE also contributed to neuroscience studies looking at brain plasticity.

DANCE conducted a human imaging (fMRI) study investigating the relationship between movement features, obtained from automated computational analyses of video clips, and the brain activity during perception of the videos.

The results showed that features requiring low-level computation, such as movement speed, map to specific brain areas related to early visual and motion sensitive regions. While mid-level features, such as expressive fluidity, are related to dynamic aspects of posture encoded in different brain areas.

As Dr. Camurri summarises, "Our computational feature based analysis suggests that the neural mechanism of movement encoding uses features that have a different neural basis from semantic categories. That is, movement perception is organized in the brain not so much by semantic categories, than by features of the body movements themselves."

Choreographing the next steps

The team developed a technology platform alongside freely available software libraries. These have been adopted for several public events, scientific experiments and mobile applications. A publicly available movement dataset was also created along with prototypes of applications for both blind and non-blind people.

The project's main public event entitled, "Atlante del gesto Genova' included the participation of blind and sighted people, with an institute for blind people, which took place from November 2016 to March 2017 and has created an on-going community of over 150 participants (including non-specialist members of the public).

The work can be adapted to generate creative content for therapy and rehabilitation. For example, a new joint laboratory with the Gaslini children's hospital in Italy was created during DANCE. Here, physiotherapists alongside computer engineers and researchers, collaborated in the creation of gaming for rehabilitation.

DANCE has also been a rich source of inspiration for further research directions such as neuroscientific studies seeking to reveal the underlying involved in the perception of high-level, affective communication through , with the aim of creating generation of novel interfaces and applications.

Explore further: Dan­cer's brains dis­play brain fre­quen­cies linked to emo­tion and memory pro­cesses

Related Stories

Dan­cer's brains dis­play brain fre­quen­cies linked to emo­tion and memory pro­cesses

May 4, 2018
Neuroscience has studied music for decades, and it has been found to activate both the cortical and deeper brain areas. Neuroscience of dance, instead, is a young but quickly growing field.

Dance aids healthier ageing

April 5, 2018
Queensland Ballet and QUT today released the results of a joint project examining the health and wellbeing benefits of ballet for older Australians.

Research suggests that the arts might play a role in empathy training

November 17, 2016
A recent study from researchers at City, University of London and the University of the Balearic Islands suggests that dancers are more emotionally sensitive than the rest of us.

How honeybees read the waggle dance

October 9, 2017
Neurons that enable honeybees to sense the waggle dance—a form of symbolic communication used by female bees to inform the hivemates about the location of a food source—are investigated in new research published in Journal ...

Brains of frequent dance spectators exhibit motor mirroring while watching familiar dance

March 21, 2012
Experienced ballet spectators with no physical expertise in ballet showed enhanced muscle-specific motor responses when watching live ballet, according to a Mar. 21 report in the open access journal PLoS ONE.

Recommended for you

Protein droplets keep neurons at the ready and immune system in balance

August 15, 2018
Inside cells, where DNA is packed tightly in the nucleus and rigid proteins keep intricate transport systems on track, some molecules have a simpler way of establishing order. They can self-organize, find one another in crowded ...

Self-control develops gradually in adolescent brain

August 15, 2018
Different parts of the brain mature at different times, which may help to explain impulsive behaviors in adolescence, suggest researchers from Penn State and the University of Pittsburgh.

Research reveals that what we see is not always what we get

August 15, 2018
Researchers are helping to explain why some people anticipate and react to fast-moving objects much quicker than others.

New approach to treating chronic itch

August 15, 2018
Researchers at the University of Zurich have discovered a new approach to suppressing itch by targeting two receptors in the spinal cord with the right experimental drug. In a series of experiments in mice and dogs, they ...

Immune cells in the brain have surprising influence on sexual behavior

August 14, 2018
Researchers have found a surprising new explanation of how young brains are shaped for sexual behavior later in life.

Scientists pinpoint brain networks responsible for naming objects

August 14, 2018
Scientists at The University of Texas Health Science Center at Houston (UTHealth) have identified the brain networks that allow you to think of an object name and then verbalize that thought. The study appeared in the July ...


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.