EyeMusic Sensory Substitution Device enables the blind to 'see' colors and shapes

February 4, 2014, IOS Press
Top: The EyeMusic's constructed audio file preserves both spatial information and image colors. The Sensory Substitution Device resizes the image to X=40 pixels (columns) by Y=24 pixels (rows) and runs a color clustering algorithm to get a 6-color image. Middle: The 2D image's X-axis is mapped to the time domain, i.e., pixels situation on the left side of the image with sound before the ones situation on its right side, illustrated here in waveform. Bottom: The Y-axis is mapped to the frequency domain, i.e., pixels situation on the upper side of the image will sound higher in frequency while those on its lower side will sound lower in frequency, illustrated here in spectrogram representation. Credit: Restorative Neurology and Neuroscience, Abboud et al.

Using auditory or tactile stimulation, Sensory Substitution Devices (SSDs) provide representations of visual information and can help the blind "see" colors and shapes. SSDs scan images and transform the information into audio or touch signals that users are trained to understand, enabling them to recognize the image without seeing it.

Currently SSDs are not widely used within the blind community because they can be cumbersome and unpleasant to use. However, a team of researchers at the Hebrew University of Jerusalem have developed the EyeMusic, a novel SSD that transmits shape and color information through a composition of pleasant musical tones, or "soundscapes." A new study published in Restorative Neurology and Neuroscience reports that using the EyeMusic SSD, both blind and blindfolded sighted participants were able to correctly identify a variety of basic shapes and colors after as little as 2-3 hours of training.

Most SSDs do not have the ability to provide color information, and some of the tactile and auditory systems used are said to be unpleasant after prolonged use. The EyeMusic, developed by senior investigator Prof. Amir Amedi, PhD, and his team at the Edmond and Lily Safra Center for Brain Sciences (ELSC) and the Institute for Medical Research Israel-Canada at the Hebrew University, scans an image and uses musical pitch to represent the location of pixels. The higher the pixel on a vertical plane, the higher the pitch of the musical note associated with it. Timing is used to indicate horizontal pixel location. Notes played closer to the opening cue represent the left side of the image, while notes played later in the sequence represent the right side. Additionally, color information is conveyed by the use of different musical instruments to create the sounds: white (vocals), blue (trumpet), red (reggae organ), green (synthesized reed), yellow (violin); black is represented by silence.

"This study is a demonstration of abilities showing that it is possible to encode the basic building blocks of shape using the EyeMusic," explains Prof. Amir Amedi. "Furthermore, the success in associating color to musical timbre holds promise for facilitating the representation of more complex shapes."

In addition to successfully identifying shapes and colors, users in the new EyeMusic study indicated they found the SSD's soundscapes to be relatively pleasant and potentially tolerable for prolonged use. "In soundscapes generated from images," notes Prof. Amedi, "there is a tendency for adjacent frequencies to be played together. Using a semitone western scale would then generate sounds that are perceived as highly dissonant. Therefore, to generate more pleasant soundscapes, we used the pentatonic musical scale that generates less dissonance when adjacent notes are played together."

While this new study shows that the EyeMusic can enable the visually impaired to extract visual shape and color information using auditory soundscapes of objects, researchers feel that this device also holds great promise for the field of visual rehabilitation in general. By providing additional color information, the EyeMusic can help facilitate object recognition and scene segmentation, while the pleasant soundscapes offer the potential of prolonged use.

"There is evidence suggesting that the brain is organized as a task-machine and not as a sensory machine. This strengthens the view that SSDs can be useful for visual rehabilitation, and therefore we suggest that the time may be ripe for turning part of the SSD spotlight back on practical visual rehabilitation," Prof. Amedi adds. "In the future, it would be intriguing to test whether the use of naturalistic sounds, like music and human voice, can facilitate learning and brain processing relying on the developed neural networks for music and human voice processing."

Additionally, the researchers hope the EyeMusic can become a tool for future neuroscience research. "It would be intriguing to explore the plastic changes associated with learning to decode color information for auditory timbre in the congenitally blind, who never experience color in their life. The utilization of the EyeMusic and its added color information in the field of neuroscience could facilitate exploring several questions in the blind with the potential to expand our understanding of brain organization in general," concludes Prof. Amedi.

Explore further: Device converting images into music helps individuals without vision reach for objects in space

More information: "EyeMusic: Introducing a "visual" colorful experience for the blind using auditory sensory substitution," by S. Abboud, S. Hanassy, S. Levy-Tzedek, S. Maidenbaum, A. Amedi. Restorative Neurology and Neuroscience, 2013. DOI: 10.3233/RNN-130338

A demonstration, "EyeMusic: Hearing colored shapes" is available from the AppStore.

Related Stories

Device converting images into music helps individuals without vision reach for objects in space

July 5, 2012
Sensory substitution devices (SSDs) use sound or touch to help the visually impaired perceive the visual scene surrounding them. The ideal SSD would assist not only in sensing the environment but also in performing daily ...

Activating the 'mind's eye': Scientists teach blind to read, recognize objects with sounds (w/ Video)

November 7, 2012
Common wisdom has it that if the visual cortex in the brain is deprived of visual information in early infanthood, it may never develop properly its functional specialization, making sight restoration later in life almost ...

How blind can 'read' shown in new research

May 16, 2012
A method developed at the Hebrew University of Jerusalem for training blind persons to "see" through the use of a sensory substitution device (SSD) has enabled those using the system to actually "read" an eye chart with letter ...

Musical brain-reading sheds light on neural processing of music

December 17, 2013
Finnish and Danish researchers have developed a new method that performs decoding, or brain-reading, during continuous listening to real music. Based on recorded brain responses, the method predicts how certain features related ...

Love of musical harmony is not nature but nurture

February 14, 2013
Our love of music and appreciation of musical harmony is learnt and not based on natural ability – a new study by University of Melbourne researchers has found.

Researchers explore how prior knowledge influences our visual senses

November 1, 2013
(Medical Xpress)—The perception and processing of color has fascinated neuroscientists for a long time, as our brain influences our perception of it to such a degree that colors could be called an illusion. One mystery ...

Recommended for you

Research reveals atomic-level changes in ALS-linked protein

January 18, 2018
For the first time, researchers have described atom-by-atom changes in a family of proteins linked to amyotrophic lateral sclerosis (ALS), a group of brain disorders known as frontotemporal dementia and degenerative diseases ...

Fragile X finding shows normal neurons that interact poorly

January 18, 2018
Neurons in mice afflicted with the genetic defect that causes Fragile X syndrome (FXS) appear similar to those in healthy mice, but these neurons fail to interact normally, resulting in the long-known cognitive impairments, ...

How your brain remembers what you had for dinner last night

January 17, 2018
Confirming earlier computational models, researchers at University of California San Diego and UC San Diego School of Medicine, with colleagues in Arizona and Louisiana, report that episodic memories are encoded in the hippocampus ...

Recording a thought's fleeting trip through the brain

January 17, 2018
University of California, Berkeley neuroscientists have tracked the progress of a thought through the brain, showing clearly how the prefrontal cortex at the front of the brain coordinates activity to help us act in response ...

Midbrain 'start neurons' control whether we walk or run

January 17, 2018
Locomotion comprises the most fundamental movements we perform. It is a complex sequence from initiating the first step, to stopping when we reach our goal. At the same time, locomotion is executed at different speeds to ...

Miles Davis is not Mozart: The brains of jazz and classical pianists work differently

January 16, 2018
Keith Jarret, world-famous jazz pianist, once answered in an interview when asked if he would ever be interested in doing a concert where he would play both jazz and classical music: "No, that's hilarious. [...] It's like ...

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.