Beyond bananas: Scientists harness 'mind reading' technology to decode complex thoughts

June 26, 2017 by Shilo Rea, Carnegie Mellon University
This latest research led by CMU's Marcel Just builds on the pioneering use of machine learning algorithms with brain imaging technology to "mind read." The findings indicate that the mind's building blocks for constructing complex thoughts are formed by the brain's various sub-systems and are not word-based. Credit: Carnegie Mellon University

Carnegie Mellon University scientists can now use brain activation patterns to identify complex thoughts, such as, "The witness shouted during the trial."

This latest research led by CMU's Marcel Just builds on the pioneering use of machine learning algorithms with imaging technology to "mind read." The findings indicate that the mind's building blocks for constructing complex thoughts are formed by the brain's various sub-systems and are not word-based. Published in Human Brain Mapping and funded by the Intelligence Advanced Research Projects Activity (IARPA), the study offers new evidence that the neural dimensions of concept representation are universal across people and languages.

"One of the big advances of the human brain was the ability to combine individual concepts into complex thoughts, to think not just of 'bananas,' but 'I like to eat bananas in evening with my friends,'" said Just, the D.O. Hebb University Professor of Psychology in the Dietrich College of Humanities and Social Sciences. "We have finally developed a way to see thoughts of that complexity in the fMRI signal. The discovery of this correspondence between thoughts and brain activation patterns tells us what the thoughts are built of."

Previous work by Just and his team showed that thoughts of familiar objects, like bananas or hammers, evoke activation patterns that involve the neural systems that we use to deal with those objects. For example, how you interact with a banana involves how you hold it, how you bite it and what it looks like.

The new study demonstrates that the brain's coding of 240 complex events, sentences like the shouting during the trial scenario uses an alphabet of 42 meaning components, or neurally plausible semantic features, consisting of features, like person, setting, size, social interaction and physical action. Each type of information is processed in a different brain system—which is how the brain also processes the information for objects. By measuring the activation in each brain system, the program can tell what types of thoughts are being contemplated.

For seven adult participants, the researchers used a computational model to assess how the for 239 sentences corresponded to the neurally plausible semantic features that characterized each sentence. Then the program was able to decode the features of the 240th left-out sentence. They went through leaving out each of the 240 sentences in turn, in what is called cross-validation.

The model was able to predict the features of the left-out sentence, with 87 percent accuracy, despite never being exposed to its activation before. It was also able to work in the other direction, to predict the activation pattern of a previously unseen sentence, knowing only its semantic features.

"Our method overcomes the unfortunate property of fMRI to smear together the signals emanating from brain events that occur close together in time, like the reading of two successive words in a sentence," Just said. "This advance makes it possible for the first time to decode thoughts containing several concepts. That's what most human thoughts are composed of."

He added, "A next step might be to decode the general type of topic a person is thinking about, such as geology or skateboarding. We are on the way to making a map of all the types of knowledge in the brain."

CMU's Jing Wang and Vladimir L. Cherkassky also participated in the study.

Discovering how the brain decodes complex thoughts is one of the many brain research breakthroughs to happen at Carnegie Mellon. CMU has created some of the first cognitive tutors, helped to develop the Jeopardy-winning Watson, founded a groundbreaking doctoral program in neural computation, and is the birthplace of artificial intelligence and cognitive psychology. Building on its strengths in biology, computer science, psychology, statistics and engineering, CMU launched BrainHub, an initiative that focuses on how the structure and activity of the brain give rise to complex behaviors.

Explore further: Brain 'reads' sentences the same in English and Portuguese

More information: Human Brain Mapping , www.ccbi.cmu.edu/reprints/Wang … Journal-preprint.pdf

Related Stories

Brain 'reads' sentences the same in English and Portuguese

November 3, 2016
An international research team led by Carnegie Mellon University has found that when the brain "reads" or decodes a sentence in English or Portuguese, its neural activation patterns are the same.

Scientists discover how the brain repurposes itself to learn scientific concepts

April 12, 2016
The human brain was initially used for basic survival tasks, such as staying safe and hunting and gathering. Yet, 200,000 years later, the same human brain is able to learn abstract concepts, like momentum, energy and gravity, ...

Brain representations of social thoughts accurately predict autism diagnosis

December 2, 2014
Psychiatric disorders—including autism—are characterized and diagnosed based on a clinical assessment of verbal and physical behavior. However, brain imaging and cognitive neuroscience are poised to provide a powerful ...

This is your brain on sentences

August 15, 2016
Researchers at the University of Rochester have, for the first time, decoded and predicted the brain activity patterns of word meanings within sentences, and successfully predicted what the brain patterns would be for new ...

Take a look, and you'll see, into your imagination

May 31, 2017
Scanning your brain to decode the contents of your mind has been a subject of intense research interest for some time. As studies have progressed, scientists have gradually been able to interpret what test subjects see, remember, ...

Recommended for you

Forty percent of people have a fictional first memory, says study

July 17, 2018
Researchers have conducted one of the largest surveys of people's first memories, finding that nearly 40 per cent of people had a first memory which is fictional.

Protein found to be key component in irregularly excited brain cells

July 17, 2018
In a new study in mice, researchers have identified a key protein involved in the irregular brain cell activity seen in autism spectrum disorders and epilepsy. The protein, p53, is well-known in cancer biology as a tumor ...

Insight without incision: Advances in noninvasive brain imaging offers improvements to epilepsy surgery

July 17, 2018
About a third of epilepsy sufferers require treatment through surgery. To check for severe epilepsy, clinicians use a surgical procedure called electrocorticography (ECoG). An ECoG maps a section of brain tissue to help clinicians ...

New drug target for remyelination in MS is identified

July 17, 2018
Remyelination, the spontaneous regeneration of the fatty insulator in the brain that keeps neurons communicating, has long been seen as crucial to the next big advance in treating multiple sclerosis (MS). However, a lack ...

Artificial neural networks now able to help reveal a brain's structure

July 17, 2018
The function of the brain is based on the connections between nerve cells. In order to map these connections and to create the connectome, the "wiring diagram" of a brain, neurobiologists capture images of the brain with ...

Convergence of synaptic signals is mediated by a protein critical for learning and memory

July 16, 2018
Inside the brain, is a complex symphony of perfectly coordinated signaling. Hundreds of different molecules amplify, modify and carry information from tiny synaptic compartments all the way through the entire length of a ...

1 comment

Adjust slider to filter visible comments by rank

Display comments: newest first

Spaced out Engineer
not rated yet Jun 26, 2017
This technology may already exist and is being used against thousands of Americans for torture.
http://www.bigger...den.com/
There is something like 300 degrees of freedom for any given word, so the order of magnitude presented here makes sense. For other regions without clear delineation and functional partitioning, multiple realizability maybe the saving grace, from those who take a connnectionalist stance.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.