Sensory substitution illuminates cognitive processes while offering therapeutic applications

June 11, 2018, CORDIS
Sensory substitution illuminates cognitive processes while offering therapeutic applications
Credit: Monstar Studio, Shutterstock

What are the implications of being able to "listen to a choreography" or "feel a ballet?" Despite the importance of observing the bodily movements of others to understand and predict actions, little is known about the underlying plasticity of the neural mechanism.

The blending of the arts and sciences, traditionally viewed as being somewhat at cross-purposes, has found fertile ground in areas such as education, therapy and rehabilitation. The EU-funded DANCE project found this to be especially true when combined with human-centred computing innovations.

The DANCE project explored how bodily movement can be expressed and perceived in auditory terms. The project developed technologies which measured the expressive qualities of full-body movement in real-time, translating gestures from visual to auditory phenomena. While contributing to brain-imaging which maps the visual and auditory cortex of blind and sighted people, the approach also offers therapeutic potential, as well as novel human-machine non-verbal interfaces.

Seeing through listening

Research has shown that the sonification (using non-speech audio to convey information) of movement can increase body awareness, along with heightened sensitivity to emotions. This finding has proven useful when applied to therapeutic endeavours.

DANCE brought together performance artists, sound designers, vision and social neuroscientists, computer scientists and engineers to further explore the potential for rehabilitation and increased social inclusion emerging from the sharing of spaces and emotions between visually impaired and sighted people. The EU-funded DANCE team have previously adopted the approach to help patients with Parkinson's disease and those suffering from chronic pain.

In the project's work, choreographed expressive movement actually creates sonic content in real time. As project coordinator Dr. Antonio Camurri, explains, "This means that itself is then conceived as a musical composition, changing its traditional dimension to offer a listening experience rather than principally a visual one."

An overarching theme of the project was to explore the way in which multisensory information runs the risks of creating sensory overload in perceivers. "So instead of augmenting the senses, with the consequent risk of blunting the experience, DANCE proceeded by deliberately creating sensory deprivation as a means of sharpening the senses." Dr. Camurri elaborates. In doing so DANCE also contributed to neuroscience studies looking at brain plasticity.

DANCE conducted a human imaging (fMRI) study investigating the relationship between movement features, obtained from automated computational analyses of video clips, and the brain activity during perception of the videos.

The results showed that features requiring low-level computation, such as movement speed, map to specific brain areas related to early visual and motion sensitive regions. While mid-level features, such as expressive fluidity, are related to dynamic aspects of posture encoded in different brain areas.

As Dr. Camurri summarises, "Our computational feature based analysis suggests that the neural mechanism of movement encoding uses features that have a different neural basis from semantic categories. That is, movement perception is organized in the brain not so much by semantic categories, than by features of the body movements themselves."

Choreographing the next steps

The team developed a technology platform alongside freely available software libraries. These have been adopted for several public events, scientific experiments and mobile applications. A publicly available movement dataset was also created along with prototypes of applications for both blind and non-blind people.

The project's main public event entitled, "Atlante del gesto Genova' included the participation of blind and sighted people, with an institute for blind people, which took place from November 2016 to March 2017 and has created an on-going community of over 150 participants (including non-specialist members of the public).

The work can be adapted to generate creative content for therapy and rehabilitation. For example, a new joint laboratory with the Gaslini children's hospital in Italy was created during DANCE. Here, physiotherapists alongside computer engineers and researchers, collaborated in the creation of gaming for rehabilitation.

DANCE has also been a rich source of inspiration for further research directions such as neuroscientific studies seeking to reveal the underlying involved in the perception of high-level, affective communication through , with the aim of creating generation of novel interfaces and applications.

Explore further: Dan­cer's brains dis­play brain fre­quen­cies linked to emo­tion and memory pro­cesses

Related Stories

Dan­cer's brains dis­play brain fre­quen­cies linked to emo­tion and memory pro­cesses

May 4, 2018
Neuroscience has studied music for decades, and it has been found to activate both the cortical and deeper brain areas. Neuroscience of dance, instead, is a young but quickly growing field.

Dance aids healthier ageing

April 5, 2018
Queensland Ballet and QUT today released the results of a joint project examining the health and wellbeing benefits of ballet for older Australians.

Research suggests that the arts might play a role in empathy training

November 17, 2016
A recent study from researchers at City, University of London and the University of the Balearic Islands suggests that dancers are more emotionally sensitive than the rest of us.

How honeybees read the waggle dance

October 9, 2017
Neurons that enable honeybees to sense the waggle dance—a form of symbolic communication used by female bees to inform the hivemates about the location of a food source—are investigated in new research published in Journal ...

Brains of frequent dance spectators exhibit motor mirroring while watching familiar dance

March 21, 2012
Experienced ballet spectators with no physical expertise in ballet showed enhanced muscle-specific motor responses when watching live ballet, according to a Mar. 21 report in the open access journal PLoS ONE.

Recommended for you

New study finds 'timing cells' in the brain may underlie an animal's inner clock

October 23, 2018
Are you taking your time when feeding your pet? Fluffy and Fido are on to you—and they can tell when you are dawdling.

Neurons reliably respond to straight lines

October 23, 2018
Single neurons in the brain's primary visual cortex can reliably detect straight lines, even though the cellular makeup of the neurons is constantly changing, according to a new study by Carnegie Mellon University neuroscientists, ...

Scientists reveal new details of how a naturally occurring hormone can boost memory in aging mice

October 23, 2018
A Columbia study in mice has revealed new details of how a naturally occurring bone hormone reverses memory loss in the aging brain. These findings about the hormone, called osteocalcin, stand to spur further investigations ...

Mutation in common protein triggers tangles, chaos inside brain cells

October 23, 2018
A pioneer in the study of neural cells revealed today (Oct. 23, 2018) how a single mutation affecting the most common protein in a supporting brain cell produces devastating, fibrous globs. These, in turn, disturb the location ...

Nerve-on-a-chip platform makes neuroprosthetics more effective

October 23, 2018
EPFL scientists have developed a miniaturized electronic platform for the stimulation and recording of peripheral nerve fibers on a chip. By modulating and rapidly recording nerve activity with a high signal-to-noise ratio, ...

The smell of lavender is relaxing, science confirms

October 23, 2018
Lavender works its relaxing magic all around us: from garden borders to bath bombs to fabric softener. But why not in our hospitals and clinics? And what is the science behind the magic?


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.