Device converting images into music helps individuals without vision reach for objects in space
Left: This is an illustration of the EyeMusic SSD, showing a user with a camera mounted on the glasses, and scalp headphones, hearing musical notes that create a mental image of the visual scene in front of him. He is reaching for the red apple in a pile of green ones. Top right: This is a close-up of the glasses-mounted camera and headphones; Bottom right: This is the hand-held camera pointed at the object of interest. Credit: Maxim Dupliy, Amir Amedi and Shelly Levy-Tzedek
Sensory substitution devices (SSDs) use sound or touch to help the visually impaired perceive the visual scene surrounding them. The ideal SSD would assist not only in sensing the environment but also in performing daily activities based on this input. For example, accurately reaching for a coffee cup, or shaking a friend's hand. In a new study, scientists trained blindfolded sighted participants to perform fast and accurate movements using a new SSD, called EyeMusic. Their results are published in the July issue of Restorative Neurology and Neuroscience.
The EyeMusic, developed by a team of researchers at the Hebrew University of Jerusalem, employs pleasant musical tones and scales to help the visually impaired "see" using music. This non-invasive SSD converts images into a combination of musical notes, or "soundscapes."
The device was developed by the senior author Prof. Amir Amedi and his team at the Edmond and Lily Safra Center for Brain Sciences (ELSC) and the Institute for Medical Research Israel-Canada at the Hebrew University. The EyeMusic scans an image and represents pixels at high vertical locations as high-pitched musical notes and low vertical locations as low-pitched notes according to a musical scale that will sound pleasant in many possible combinations. The image is scanned continuously, from left to right, and an auditory cue is used to mark the start of the scan. The horizontal location of a pixel is indicated by the timing of the musical notes relative to the cue (the later it is sounded after the cue, the farther it is to the right), and the brightness is encoded by the loudness of the sound.
The EyeMusic's algorithm uses different musical instruments for each of the five colors: white (vocals), blue (trumpet), red (reggae organ), green (synthesized reed), yellow (violin); Black is represented by silence. Prof. Amedi mentions that "The notes played span five octaves and were carefully chosen by musicians to create a pleasant experience for the users." Sample sound recordings are available at http://brain.huji.ac.il/em/.
"We demonstrated in this study that the EyeMusic, which employs pleasant musical scales to convey visual information, can be used after a short training period (in some cases, less than half an hour) to guide movements, similar to movements guided visually," explain lead investigators Drs. Shelly Levy-Tzedek, an ELSC researcher at the Faculty of Medicine, Hebrew University, Jerusalem, and Prof. Amir Amedi. "The level of accuracy reached in our study indicates that performing daily tasks with an SSD is feasible, and indicates a potential for rehabilitative use."
The study tested the ability of 18 blindfolded sighted individuals to perform movements guided by the EyeMusic, and compared those movements to those performed with visual guidance. At first, the blindfolded participants underwent a short familiarization session, where they learned to identify the location of a single object (a white square) or of two adjacent objects (a white and a blue square).
In the test sessions, participants used a stylus on a digitizing tablet to point to a white square located either in the north, the south, the east or the west. In one block of trials they were blindfolded (SSD block), and in the other block (VIS block) the arm was placed under an opaque cover, so they could see the screen but did not have direct visual feedback from the hand. The endpoint location of their hand was marked by a blue square. In the SSD block, they received feedback via the EyeMusic. In the VIS block, the feedback was visual.
"Participants were able to use auditory information to create a relatively precise spatial representation," notes Dr. Levy-Tzedek.
The study lends support to the hypothesis that representation of space in the brain may not be dependent on the modality with which the spatial information is received, and that very little training is required to create a representation of space without vision, using sounds to guide fast and accurate movements. "SSDs may have great potential to provide detailed spatial information for the visually impaired, allowing them to interact with their external environment and successfully make movements based on this information, but further research is now required to evaluate the use of our device in the blind " concludes Dr. Levy-Tzedek. These results demonstrate the potential application of the EyeMusic in performing everyday tasks from accurately reaching for the red (but not the green!) apples in the produce aisle, to, perhaps one day, playing a Kinect / Xbox game.
More information: Fast, Accurate Reaching Movements with a Visual-to-Auditory Sensory Substitution Device, by S. Levy-Tzedek, S. Hanassy, S. Abboud, S. Maidenbaum, A. Amedi. Restorative Neurology and Neuroscience, 30: 4 (July 2012). DOI: 10.3233/RNN-2012-110219
Provided by IOS Press
- How blind can 'read' shown in new research May 16, 2012 | not rated yet | 0
- See no shape, touch no shape, hear a shape? Oct 18, 2010 | not rated yet | 0
- Brain doesn't need vision at all in order to 'read' material Feb 22, 2011 | not rated yet | 0
- Musical illusion fools audiences and performers, says researcher Jan 05, 2010 | not rated yet | 0
- Sensitivity of brain center for 'sound space' defined Sep 20, 2007 | not rated yet | 0
- Motion perception revisited: High Phi effect challenges established motion perception assumptions Apr 23, 2013 | 3 / 5 (2) | 2
- Anything you can do I can do better: Neuromolecular foundations of the superiority illusion (Update) Apr 02, 2013 | 4.5 / 5 (11) | 5
- The visual system as economist: Neural resource allocation in visual adaptation Mar 30, 2013 | 5 / 5 (2) | 9
- Separate lives: Neuronal and organismal lifespans decoupled Mar 27, 2013 | 4.9 / 5 (8) | 0
- Sizing things up: The evolutionary neurobiology of scale invariance Feb 28, 2013 | 4.8 / 5 (10) | 14
Why is zone 1 in liver more prone to ischemic injury?
May 23, 2013 Hi, Is it because around central vein, there is only deoxygenated blood from the vein where as in the periphery there is hepatic artery. Also why...
How can there be villous adenoma in colon, if there are no villi there
May 22, 2013 As title suggest. Thanks :smile:
How can there be a term called "intestinal metaplasia" of stomach
May 21, 2013 Hello everyone, Ok Stomach's normal epithelium is simple columnar, now in intestinal type of adenocarcinoma of stomach it undergoes "intestinal...
Pressure-volume curve: Elastic Recoil Pressure don't make sense
May 18, 2013 From pressure-volume curve of the lung and chest wall (attached photo), I don't understand why would the elastic recoil pressure of the lung is...
If you became brain-dead, would you want them to pull the plug?
May 17, 2013 I'd want the rest of me to stay alive. Sure it's a lousy way to live but it beats being all-the-way dead. Maybe if I make it 20 years they'll...
MRI bill question
May 15, 2013 Dear PFers, The hospital gave us a $12k bill for one MRI (head with contrast). The people I talked to at the hospital tell me that they do not...
- More from Physics Forums - Medical Sciences
More news stories
(Medical Xpress)—Regulating the distribution of power in neurons is done by a system that makes the national electric grid look simple by comparison. Each neuron has several thousand mitochondria confined ...
Neuroscience 11 hours ago | 4.8 / 5 (6) | 0 |
(Medical Xpress)—The human brain is able to identify individuals' voices by comparing them against an internal 'average voice' prototype, according to neuroscientists.
Neuroscience 15 hours ago | 1 / 5 (1) | 0 |
A new study determined that children and adolescents with seizures involving the temporal lobe are likely to have clinically significant behavioral problems and psychiatric illness, especially depression. Findings published ...
Neuroscience 15 hours ago | not rated yet | 0
As the human body fine-tunes its neurological wiring, nerve cells often must fix a faulty connection by amputating an axon—the "business end" of the neuron that sends electrical impulses to tissues or other ...
Neuroscience 17 hours ago | 5 / 5 (2) | 0 |
A study by researchers at Henry Ford Hospital found "substantial evidence" that a regenerative process involving damaged nerve fibers in the spinal cord could hold the key to better functional recovery by most stroke victims.
Neuroscience 17 hours ago | not rated yet | 0 |
The British Menopause Society and Women's Health Concern have today released updated guidelines on Hormone Replacement Therapy (HRT) to provide clarity around the role of HRT, the benefits and the risks. The new guidelines ...
45 minutes ago | not rated yet | 0
A brief visual task can predict IQ, according to a new study. This surprisingly simple exercise measures the brain's unconscious ability to filter out visual movement. The study shows that individuals whose ...
17 hours ago | 4.5 / 5 (10) | 1 |
Teams of highly respected Alzheimer's researchers failed to replicate what appeared to be breakthrough results for the treatment of this brain disease when they were published last year in the journal Science.
15 hours ago | 5 / 5 (1) | 2 |
Scientists at the National Institutes of Health report they have discovered in mouse studies that a small molecule released in the spinal cord triggers a process that is later experienced in the brain as ...
15 hours ago | 5 / 5 (2) | 0 |
Little is known about why asthma develops, how it constricts the airway or why response to treatments varies between patients. Now, a team of researchers at Weill Cornell Medical College, Columbia University Medical Center ...
15 hours ago | not rated yet | 0 |
Ethnic background plays a surprisingly large role in how diabetes develops on a cellular level, according to two new studies led by researchers at the Stanford University School of Medicine.
13 hours ago | not rated yet | 0 |