Training computers to understand the human brain
The activation maps of the two contrasts (hot color: mammal > tool ; cool color: tool > mammal) computed from the 10 datasets of our participants.
Understanding how the human brain categorizes information through signs and language is a key part of developing computers that can 'think' and 'see' in the same way as humans. Hiroyuki Akama at the Graduate School of Decision Science and Technology, Tokyo Institute of Technology, together with co-workers in Yokohama, the USA, Italy and the UK, have completed a study using fMRI datasets to train a computer to predict the semantic category of an image originally viewed by five different people.
The participants were asked to look at pictures of animals and hand tools together with an auditory or written (orthographic) description. They were asked to silently 'label' each pictured object with certain properties, whilst undergoing an fMRI brain scan. The resulting scans were analysed using algorithms that identified patterns relating to the two separate semantic groups (animal or tool).
After 'training' the algorithms in this way using some of the auditory session data, the computer correctly identified the remaining scans 80-90% of the time. Similar results were obtained with the orthographic session data. A cross-modal approach, namely training the computer using auditory data but testing it using orthographic, reduced performance to 65-75%. Continued research in this area could lead to systems that allow people to speak through a computer simply by thinking about what they want to say.
Understanding how the human brain categorizes information through signs and language is a key part of developing computers that can 'think' and 'see' in the same way as humans. It is only in recent years that the field of semantics has been explored through the analysis of brain scans and brain activity in response to both language-based and visual inputs. Teaching computers to read brain scans and interpret the language encoded in brain activity could have a variety of uses in medical science and artificial intelligence.
Now, Hiroyuki Akama at the Graduate School of Decision Science and Technology, Tokyo Institute of Technology, together with co-workers in Yokohama, the USA, Italy and the UK, have completed a study using fMRI datasets to train a computer to predict the semantic category of an image originally viewed by five different people.
The five participants in the project were shown two sets of forty randomly arranged pictures during the experiment. The pictures came from two distinct categories – either an animal, or a hand tool. In the first session, twenty images of animals and twenty of hand tools were accompanied by the spoken Japanese name of each object (auditory). In the second session - shown to the participants several days later - the same twenty randomly ordered images were accompanied by Japanese written characters (orthography). Each participant was asked to silently 'label' each image with properties they associate with that object in their mind.
During each session, the participants were scanned using fMRI technology. This provided Akama and his team with 240 individual scans showing brain activity for each session. The researchers analyzed the brain scans using a technique called multi-voxel pattern analysis (MVPA). This involves using computer algorithms to identify repeating patterns of brain activity across voxels, the cube-shaped elements that make up the 3D scan images. Interestingly, animal pictures tended to induce activity in the visual part of the brain, whereas tool pictures triggered a response more from sensory-motor areas – a phenomenon reported in previous studies.
The MVPA results were then used to find out if the computer could predict whether or not the participants were viewing an animal or hand tool image by looking at the patterns in the scans.
Several different tests were given to the computer. After training the machine to recognise patterns related to 'animals' and 'tools' in some of the auditory session data for example, the computer correctly identified the remaining auditory data scans as animal or tool 80-90% of the time. The computer found the auditory data easier to predict, although it had a very similar success rate when identifying the orthographic session data.
Akama and his team then decided to try a cross-modal approach, namely training the computer using one session data set but testing it using the other. As perhaps would be expected, the brain scans for auditory and orthographic sessions differed, as people think in different ways when listening and reading. However, the computer suffered an even stronger performance penalty than anticipated, with success rates down to 65-75%. The exact reasons for this are unclear, although the researchers point to a combination of timing differences (the time taken for the participants to respond to written as opposed to auditory information) and spatial differences (the anatomy of the individuals' brains differing slightly and thereby affecting the voxel distributions).
One future application of experiments such as this could be the development of real-time brain-computer-interfaces. Such devices could allow patients with communication impairments to speak through a computer simply by thinking about what they want to say.
More information: H. Akama et al. Decoding semantics across fMRI sessions with different stimulus modalities: a practical MVPA study. Frontiers in Neuroinformatics 6 (24) (2012). doi: 10.3389/fninf.2012.00024
Provided by Tokyo Institute of Technology
- Can You Read My Mind? Mar 23, 2005 | not rated yet | 0
- Scientists probe connection between sight and touch in the brain Sep 08, 2011 | not rated yet | 0
- Carnegie Mellon study identifies where thoughts of familiar objects occur inside the human brain Jan 02, 2008 | not rated yet | 0
- Word association: Study matches brain scans with complex thought Aug 31, 2011 | not rated yet | 0
- In the brain, winning is everywhere Oct 05, 2011 | not rated yet | 0
- Motion perception revisited: High Phi effect challenges established motion perception assumptions Apr 23, 2013 | 3 / 5 (2) | 2
- Anything you can do I can do better: Neuromolecular foundations of the superiority illusion (Update) Apr 02, 2013 | 4.5 / 5 (11) | 5
- The visual system as economist: Neural resource allocation in visual adaptation Mar 30, 2013 | 5 / 5 (2) | 9
- Separate lives: Neuronal and organismal lifespans decoupled Mar 27, 2013 | 4.9 / 5 (8) | 0
- Sizing things up: The evolutionary neurobiology of scale invariance Feb 28, 2013 | 4.8 / 5 (10) | 14
Why is zone 1 in liver more prone to ischemic injury?
May 23, 2013 Hi, Is it because around central vein, there is only deoxygenated blood from the vein where as in the periphery there is hepatic artery. Also why...
How can there be villous adenoma in colon, if there are no villi there
May 22, 2013 As title suggest. Thanks :smile:
How can there be a term called "intestinal metaplasia" of stomach
May 21, 2013 Hello everyone, Ok Stomach's normal epithelium is simple columnar, now in intestinal type of adenocarcinoma of stomach it undergoes "intestinal...
Pressure-volume curve: Elastic Recoil Pressure don't make sense
May 18, 2013 From pressure-volume curve of the lung and chest wall (attached photo), I don't understand why would the elastic recoil pressure of the lung is...
If you became brain-dead, would you want them to pull the plug?
May 17, 2013 I'd want the rest of me to stay alive. Sure it's a lousy way to live but it beats being all-the-way dead. Maybe if I make it 20 years they'll...
MRI bill question
May 15, 2013 Dear PFers, The hospital gave us a $12k bill for one MRI (head with contrast). The people I talked to at the hospital tell me that they do not...
- More from Physics Forums - Medical Sciences
More news stories
New research presented today shows that formation of new neurons in the hippocampus - a brain region known for its importance in learning and remembering - could cause forgetting of old memories by causing a reorganization ...
Neuroscience May 24, 2013 | 4 / 5 (4) | 0
How can healthy people who hear voices help schizophrenics? Finding the answer for this is at the centre of research conducted at the University of Bergen.
Neuroscience May 24, 2013 | 4 / 5 (2) | 2
One of the major frontiers of modern science is a comprehensive understanding of the human brain and its functions to guide the development of new technologies in information and communication. In a major announcement for ...
Neuroscience May 24, 2013 | 3.7 / 5 (3) | 0
(Medical Xpress)—Regulating the distribution of power in neurons is done by a system that makes the national electric grid look simple by comparison. Each neuron has several thousand mitochondria confined ...
Neuroscience May 23, 2013 | 4.9 / 5 (10) | 0 |
(Medical Xpress)—The human brain is able to identify individuals' voices by comparing them against an internal 'average voice' prototype, according to neuroscientists.
Neuroscience May 23, 2013 | 3.7 / 5 (3) | 3 |
Coenzyme Q10 decreases all cause mortality by half, according to the results of a multicentre randomised double blind trial presented today at Heart Failure 2013 congress. It is the first drug to improve heart failure mortality ...
12 hours ago | 5 / 5 (2) | 5
(HealthDay)—Animals make great companions for senior citizens, but elderly people who always drive with a pet in the car are far more likely to crash than those who never drive with a pet, researchers have ...
4 hours ago | not rated yet | 0
Heart failure accelerates the aging process and brings on early andropausal syndrome (AS), according to research presented today at the Heart Failure Congress 2013. AS, also referred to as male 'menopause', was four times ...
12 hours ago | not rated yet | 1
Mortality and length of stay are highest in heart failure patients admitted in January, on Friday, and overnight, according to research presented today at the Heart Failure Congress 2013. The analysis of nearly 1 million ...
12 hours ago | not rated yet | 0
(AP)—Department of Justice lawyers have again asked a federal appeals court in New York to delay lifting age restrictions and prescription requirements on an emergency contraceptive popularly known as the morning-after ...
12 hours ago | not rated yet | 0