Beyond bananas: Scientists harness 'mind reading' technology to decode complex thoughts

June 26, 2017 by Shilo Rea, Carnegie Mellon University
This latest research led by CMU's Marcel Just builds on the pioneering use of machine learning algorithms with brain imaging technology to "mind read." The findings indicate that the mind's building blocks for constructing complex thoughts are formed by the brain's various sub-systems and are not word-based. Credit: Carnegie Mellon University

Carnegie Mellon University scientists can now use brain activation patterns to identify complex thoughts, such as, "The witness shouted during the trial."

This latest research led by CMU's Marcel Just builds on the pioneering use of machine learning algorithms with imaging technology to "mind read." The findings indicate that the mind's building blocks for constructing complex thoughts are formed by the brain's various sub-systems and are not word-based. Published in Human Brain Mapping and funded by the Intelligence Advanced Research Projects Activity (IARPA), the study offers new evidence that the neural dimensions of concept representation are universal across people and languages.

"One of the big advances of the human brain was the ability to combine individual concepts into complex thoughts, to think not just of 'bananas,' but 'I like to eat bananas in evening with my friends,'" said Just, the D.O. Hebb University Professor of Psychology in the Dietrich College of Humanities and Social Sciences. "We have finally developed a way to see thoughts of that complexity in the fMRI signal. The discovery of this correspondence between thoughts and brain activation patterns tells us what the thoughts are built of."

Previous work by Just and his team showed that thoughts of familiar objects, like bananas or hammers, evoke activation patterns that involve the neural systems that we use to deal with those objects. For example, how you interact with a banana involves how you hold it, how you bite it and what it looks like.

The new study demonstrates that the brain's coding of 240 complex events, sentences like the shouting during the trial scenario uses an alphabet of 42 meaning components, or neurally plausible semantic features, consisting of features, like person, setting, size, social interaction and physical action. Each type of information is processed in a different brain system—which is how the brain also processes the information for objects. By measuring the activation in each brain system, the program can tell what types of thoughts are being contemplated.

For seven adult participants, the researchers used a computational model to assess how the for 239 sentences corresponded to the neurally plausible semantic features that characterized each sentence. Then the program was able to decode the features of the 240th left-out sentence. They went through leaving out each of the 240 sentences in turn, in what is called cross-validation.

The model was able to predict the features of the left-out sentence, with 87 percent accuracy, despite never being exposed to its activation before. It was also able to work in the other direction, to predict the activation pattern of a previously unseen sentence, knowing only its semantic features.

"Our method overcomes the unfortunate property of fMRI to smear together the signals emanating from brain events that occur close together in time, like the reading of two successive words in a sentence," Just said. "This advance makes it possible for the first time to decode thoughts containing several concepts. That's what most human thoughts are composed of."

He added, "A next step might be to decode the general type of topic a person is thinking about, such as geology or skateboarding. We are on the way to making a map of all the types of knowledge in the brain."

CMU's Jing Wang and Vladimir L. Cherkassky also participated in the study.

Discovering how the brain decodes complex thoughts is one of the many brain research breakthroughs to happen at Carnegie Mellon. CMU has created some of the first cognitive tutors, helped to develop the Jeopardy-winning Watson, founded a groundbreaking doctoral program in neural computation, and is the birthplace of artificial intelligence and cognitive psychology. Building on its strengths in biology, computer science, psychology, statistics and engineering, CMU launched BrainHub, an initiative that focuses on how the structure and activity of the brain give rise to complex behaviors.

Explore further: Brain 'reads' sentences the same in English and Portuguese

More information: Human Brain Mapping , www.ccbi.cmu.edu/reprints/Wang … Journal-preprint.pdf

Related Stories

Brain 'reads' sentences the same in English and Portuguese

November 3, 2016
An international research team led by Carnegie Mellon University has found that when the brain "reads" or decodes a sentence in English or Portuguese, its neural activation patterns are the same.

Scientists discover how the brain repurposes itself to learn scientific concepts

April 12, 2016
The human brain was initially used for basic survival tasks, such as staying safe and hunting and gathering. Yet, 200,000 years later, the same human brain is able to learn abstract concepts, like momentum, energy and gravity, ...

Brain representations of social thoughts accurately predict autism diagnosis

December 2, 2014
Psychiatric disorders—including autism—are characterized and diagnosed based on a clinical assessment of verbal and physical behavior. However, brain imaging and cognitive neuroscience are poised to provide a powerful ...

This is your brain on sentences

August 15, 2016
Researchers at the University of Rochester have, for the first time, decoded and predicted the brain activity patterns of word meanings within sentences, and successfully predicted what the brain patterns would be for new ...

Take a look, and you'll see, into your imagination

May 31, 2017
Scanning your brain to decode the contents of your mind has been a subject of intense research interest for some time. As studies have progressed, scientists have gradually been able to interpret what test subjects see, remember, ...

Recommended for you

How the brain makes rapid, fine adjustments in motor activity

October 18, 2018
Short-term motor learning appears not to require physical change in the brain Brain's premotor cortex may use a 'neural scratch pad' to calculate fine adjustments Brain can try different things in simulation without 'screwing ...

Electrical properties of dendrites help explain our brain's unique computing power

October 18, 2018
Neurons in the human brain receive electrical signals from thousands of other cells, and long neural extensions called dendrites play a critical role in incorporating all of that information so the cells can respond appropriately.

Brain cells called astrocytes have unexpected role in brain 'plasticity'

October 18, 2018
When we're born, our brains have a great deal of flexibility. Having this flexibility to grow and change gives the immature brain the ability to adapt to new experiences and organize its interconnecting web of neural circuits. ...

Weight loss success linked with active self-control regions of the brain

October 18, 2018
New research suggests that higher-level brain functions have a major role in losing weight. In a study among 24 participants at a weight-loss clinic, those who achieved greatest success in terms of weight loss demonstrated ...

Study pinpoints what makes human neurons unique

October 18, 2018
Human neurons are much larger than those of model organisms mice and rats, so it's been unclear whether it's size that makes a difference in our brain's computational power. Now, in a study appearing October 18 in the journal ...

New causative gene found in severe childhood epilepsy

October 18, 2018
A large international research team has discovered a new genetic cause for a severe, difficult-to-treat childhood epilepsy syndrome. Spontaneous mutations in one gene disrupt the flow of calcium in brain cells, resulting ...

1 comment

Adjust slider to filter visible comments by rank

Display comments: newest first

Spaced out Engineer
not rated yet Jun 26, 2017
This technology may already exist and is being used against thousands of Americans for torture.
http://www.bigger...den.com/
There is something like 300 degrees of freedom for any given word, so the order of magnitude presented here makes sense. For other regions without clear delineation and functional partitioning, multiple realizability maybe the saving grace, from those who take a connnectionalist stance.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.