Beyond bananas: Scientists harness 'mind reading' technology to decode complex thoughts

June 26, 2017 by Shilo Rea, Carnegie Mellon University
This latest research led by CMU's Marcel Just builds on the pioneering use of machine learning algorithms with brain imaging technology to "mind read." The findings indicate that the mind's building blocks for constructing complex thoughts are formed by the brain's various sub-systems and are not word-based. Credit: Carnegie Mellon University

Carnegie Mellon University scientists can now use brain activation patterns to identify complex thoughts, such as, "The witness shouted during the trial."

This latest research led by CMU's Marcel Just builds on the pioneering use of machine learning algorithms with imaging technology to "mind read." The findings indicate that the mind's building blocks for constructing complex thoughts are formed by the brain's various sub-systems and are not word-based. Published in Human Brain Mapping and funded by the Intelligence Advanced Research Projects Activity (IARPA), the study offers new evidence that the neural dimensions of concept representation are universal across people and languages.

"One of the big advances of the human brain was the ability to combine individual concepts into complex thoughts, to think not just of 'bananas,' but 'I like to eat bananas in evening with my friends,'" said Just, the D.O. Hebb University Professor of Psychology in the Dietrich College of Humanities and Social Sciences. "We have finally developed a way to see thoughts of that complexity in the fMRI signal. The discovery of this correspondence between thoughts and brain activation patterns tells us what the thoughts are built of."

Previous work by Just and his team showed that thoughts of familiar objects, like bananas or hammers, evoke activation patterns that involve the neural systems that we use to deal with those objects. For example, how you interact with a banana involves how you hold it, how you bite it and what it looks like.

The new study demonstrates that the brain's coding of 240 complex events, sentences like the shouting during the trial scenario uses an alphabet of 42 meaning components, or neurally plausible semantic features, consisting of features, like person, setting, size, social interaction and physical action. Each type of information is processed in a different brain system—which is how the brain also processes the information for objects. By measuring the activation in each brain system, the program can tell what types of thoughts are being contemplated.

For seven adult participants, the researchers used a computational model to assess how the for 239 sentences corresponded to the neurally plausible semantic features that characterized each sentence. Then the program was able to decode the features of the 240th left-out sentence. They went through leaving out each of the 240 sentences in turn, in what is called cross-validation.

The model was able to predict the features of the left-out sentence, with 87 percent accuracy, despite never being exposed to its activation before. It was also able to work in the other direction, to predict the activation pattern of a previously unseen sentence, knowing only its semantic features.

"Our method overcomes the unfortunate property of fMRI to smear together the signals emanating from brain events that occur close together in time, like the reading of two successive words in a sentence," Just said. "This advance makes it possible for the first time to decode thoughts containing several concepts. That's what most human thoughts are composed of."

He added, "A next step might be to decode the general type of topic a person is thinking about, such as geology or skateboarding. We are on the way to making a map of all the types of knowledge in the brain."

CMU's Jing Wang and Vladimir L. Cherkassky also participated in the study.

Discovering how the brain decodes complex thoughts is one of the many brain research breakthroughs to happen at Carnegie Mellon. CMU has created some of the first cognitive tutors, helped to develop the Jeopardy-winning Watson, founded a groundbreaking doctoral program in neural computation, and is the birthplace of artificial intelligence and cognitive psychology. Building on its strengths in biology, computer science, psychology, statistics and engineering, CMU launched BrainHub, an initiative that focuses on how the structure and activity of the brain give rise to complex behaviors.

Explore further: Brain 'reads' sentences the same in English and Portuguese

More information: Human Brain Mapping , www.ccbi.cmu.edu/reprints/Wang … Journal-preprint.pdf

Related Stories

Brain 'reads' sentences the same in English and Portuguese

November 3, 2016
An international research team led by Carnegie Mellon University has found that when the brain "reads" or decodes a sentence in English or Portuguese, its neural activation patterns are the same.

Scientists discover how the brain repurposes itself to learn scientific concepts

April 12, 2016
The human brain was initially used for basic survival tasks, such as staying safe and hunting and gathering. Yet, 200,000 years later, the same human brain is able to learn abstract concepts, like momentum, energy and gravity, ...

Brain representations of social thoughts accurately predict autism diagnosis

December 2, 2014
Psychiatric disorders—including autism—are characterized and diagnosed based on a clinical assessment of verbal and physical behavior. However, brain imaging and cognitive neuroscience are poised to provide a powerful ...

This is your brain on sentences

August 15, 2016
Researchers at the University of Rochester have, for the first time, decoded and predicted the brain activity patterns of word meanings within sentences, and successfully predicted what the brain patterns would be for new ...

Take a look, and you'll see, into your imagination

May 31, 2017
Scanning your brain to decode the contents of your mind has been a subject of intense research interest for some time. As studies have progressed, scientists have gradually been able to interpret what test subjects see, remember, ...

Recommended for you

How your brain remembers what you had for dinner last night

January 17, 2018
Confirming earlier computational models, researchers at University of California San Diego and UC San Diego School of Medicine, with colleagues in Arizona and Louisiana, report that episodic memories are encoded in the hippocampus ...

Recording a thought's fleeting trip through the brain

January 17, 2018
University of California, Berkeley neuroscientists have tracked the progress of a thought through the brain, showing clearly how the prefrontal cortex at the front of the brain coordinates activity to help us act in response ...

Midbrain 'start neurons' control whether we walk or run

January 17, 2018
Locomotion comprises the most fundamental movements we perform. It is a complex sequence from initiating the first step, to stopping when we reach our goal. At the same time, locomotion is executed at different speeds to ...

A 'touching sight': How babies' brains process touch builds foundations for learning

January 16, 2018
Touch is the first of the five senses to develop, yet scientists know far less about the baby's brain response to touch than to, say, the sight of mom's face, or the sound of her voice.

Brain zaps may help curb tics of Tourette syndrome

January 16, 2018
Electric zaps can help rewire the brains of Tourette syndrome patients, effectively reducing their uncontrollable vocal and motor tics, a new study shows.

Researchers identify protein involved in cocaine addiction

January 16, 2018
Mount Sinai researchers have identified a protein produced by the immune system—granulocyte-colony stimulating factor (G-CSF)—that could be responsible for the development of cocaine addiction.

1 comment

Adjust slider to filter visible comments by rank

Display comments: newest first

Spaced out Engineer
not rated yet Jun 26, 2017
This technology may already exist and is being used against thousands of Americans for torture.
http://www.bigger...den.com/
There is something like 300 degrees of freedom for any given word, so the order of magnitude presented here makes sense. For other regions without clear delineation and functional partitioning, multiple realizability maybe the saving grace, from those who take a connnectionalist stance.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.