Beyond bananas: Scientists harness 'mind reading' technology to decode complex thoughts

June 26, 2017 by Shilo Rea
This latest research led by CMU's Marcel Just builds on the pioneering use of machine learning algorithms with brain imaging technology to "mind read." The findings indicate that the mind's building blocks for constructing complex thoughts are formed by the brain's various sub-systems and are not word-based. Credit: Carnegie Mellon University

Carnegie Mellon University scientists can now use brain activation patterns to identify complex thoughts, such as, "The witness shouted during the trial."

This latest research led by CMU's Marcel Just builds on the pioneering use of machine learning algorithms with imaging technology to "mind read." The findings indicate that the mind's building blocks for constructing complex thoughts are formed by the brain's various sub-systems and are not word-based. Published in Human Brain Mapping and funded by the Intelligence Advanced Research Projects Activity (IARPA), the study offers new evidence that the neural dimensions of concept representation are universal across people and languages.

"One of the big advances of the human brain was the ability to combine individual concepts into complex thoughts, to think not just of 'bananas,' but 'I like to eat bananas in evening with my friends,'" said Just, the D.O. Hebb University Professor of Psychology in the Dietrich College of Humanities and Social Sciences. "We have finally developed a way to see thoughts of that complexity in the fMRI signal. The discovery of this correspondence between thoughts and brain activation patterns tells us what the thoughts are built of."

Previous work by Just and his team showed that thoughts of familiar objects, like bananas or hammers, evoke activation patterns that involve the neural systems that we use to deal with those objects. For example, how you interact with a banana involves how you hold it, how you bite it and what it looks like.

The new study demonstrates that the brain's coding of 240 complex events, sentences like the shouting during the trial scenario uses an alphabet of 42 meaning components, or neurally plausible semantic features, consisting of features, like person, setting, size, social interaction and physical action. Each type of information is processed in a different brain system—which is how the brain also processes the information for objects. By measuring the activation in each brain system, the program can tell what types of thoughts are being contemplated.

For seven adult participants, the researchers used a computational model to assess how the for 239 sentences corresponded to the neurally plausible semantic features that characterized each sentence. Then the program was able to decode the features of the 240th left-out sentence. They went through leaving out each of the 240 sentences in turn, in what is called cross-validation.

The model was able to predict the features of the left-out sentence, with 87 percent accuracy, despite never being exposed to its activation before. It was also able to work in the other direction, to predict the activation pattern of a previously unseen sentence, knowing only its semantic features.

"Our method overcomes the unfortunate property of fMRI to smear together the signals emanating from brain events that occur close together in time, like the reading of two successive words in a sentence," Just said. "This advance makes it possible for the first time to decode thoughts containing several concepts. That's what most human thoughts are composed of."

He added, "A next step might be to decode the general type of topic a person is thinking about, such as geology or skateboarding. We are on the way to making a map of all the types of knowledge in the brain."

CMU's Jing Wang and Vladimir L. Cherkassky also participated in the study.

Discovering how the brain decodes complex thoughts is one of the many brain research breakthroughs to happen at Carnegie Mellon. CMU has created some of the first cognitive tutors, helped to develop the Jeopardy-winning Watson, founded a groundbreaking doctoral program in neural computation, and is the birthplace of artificial intelligence and cognitive psychology. Building on its strengths in biology, computer science, psychology, statistics and engineering, CMU launched BrainHub, an initiative that focuses on how the structure and activity of the brain give rise to complex behaviors.

Explore further: Brain 'reads' sentences the same in English and Portuguese

More information: Human Brain Mapping , www.ccbi.cmu.edu/reprints/Wang … Journal-preprint.pdf

Related Stories

Brain 'reads' sentences the same in English and Portuguese

November 3, 2016
An international research team led by Carnegie Mellon University has found that when the brain "reads" or decodes a sentence in English or Portuguese, its neural activation patterns are the same.

Scientists discover how the brain repurposes itself to learn scientific concepts

April 12, 2016
The human brain was initially used for basic survival tasks, such as staying safe and hunting and gathering. Yet, 200,000 years later, the same human brain is able to learn abstract concepts, like momentum, energy and gravity, ...

Brain representations of social thoughts accurately predict autism diagnosis

December 2, 2014
Psychiatric disorders—including autism—are characterized and diagnosed based on a clinical assessment of verbal and physical behavior. However, brain imaging and cognitive neuroscience are poised to provide a powerful ...

This is your brain on sentences

August 15, 2016
Researchers at the University of Rochester have, for the first time, decoded and predicted the brain activity patterns of word meanings within sentences, and successfully predicted what the brain patterns would be for new ...

Take a look, and you'll see, into your imagination

May 31, 2017
Scanning your brain to decode the contents of your mind has been a subject of intense research interest for some time. As studies have progressed, scientists have gradually been able to interpret what test subjects see, remember, ...

Recommended for you

Touching helps build the sexual brain

September 21, 2017
Hormones or sexual experience? Which of these is crucial for the onset of puberty? It seems that when rats are touched on their genitals, their brain changes and puberty accelerates. In a new study publishing September 21 ...

Gene immunotherapy protects against multiple sclerosis in mice

September 21, 2017
A potent and long-lasting gene immunotherapy approach prevents and reverses symptoms of multiple sclerosis in mice, according to a study published September 21st in the journal Molecular Therapy. Multiple sclerosis is an ...

Neuron types in brain are defined by gene activity shaping their communication patterns

September 21, 2017
In a major step forward in research, scientists at Cold Spring Harbor Laboratory (CSHL) today publish in Cell a discovery about the molecular-genetic basis of neuronal cell types. Neurons are the basic building blocks that ...

Your neurons register familiar faces, whether you notice them or not

September 21, 2017
When people see an image of a person they recognize—the famous tennis player Roger Federer or actress Halle Berry, for instance—particular cells light up in the brain. Now, researchers reporting in Current Biology on ...

Highly precise wiring in the cerebral cortex

September 21, 2017
Our brains house extremely complex neuronal circuits whose detailed structures are still largely unknown. This is especially true for the cerebral cortex of mammals, where, among other things, vision, thoughts or spatial ...

Faulty cell signaling derails cerebral cortex development, could it lead to autism?

September 20, 2017
As the embryonic brain develops, an incredibly complex cascade of cellular events occur, starting with progenitors - the originating cells that generate neurons and spur proper cortex development. If this cascade malfunctions ...

1 comment

Adjust slider to filter visible comments by rank

Display comments: newest first

Spaced out Engineer
not rated yet Jun 26, 2017
This technology may already exist and is being used against thousands of Americans for torture.
http://www.bigger...den.com/
There is something like 300 degrees of freedom for any given word, so the order of magnitude presented here makes sense. For other regions without clear delineation and functional partitioning, multiple realizability maybe the saving grace, from those who take a connnectionalist stance.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.