Beyond bananas: Scientists harness 'mind reading' technology to decode complex thoughts

June 26, 2017 by Shilo Rea
This latest research led by CMU's Marcel Just builds on the pioneering use of machine learning algorithms with brain imaging technology to "mind read." The findings indicate that the mind's building blocks for constructing complex thoughts are formed by the brain's various sub-systems and are not word-based. Credit: Carnegie Mellon University

Carnegie Mellon University scientists can now use brain activation patterns to identify complex thoughts, such as, "The witness shouted during the trial."

This latest research led by CMU's Marcel Just builds on the pioneering use of machine learning algorithms with imaging technology to "mind read." The findings indicate that the mind's building blocks for constructing complex thoughts are formed by the brain's various sub-systems and are not word-based. Published in Human Brain Mapping and funded by the Intelligence Advanced Research Projects Activity (IARPA), the study offers new evidence that the neural dimensions of concept representation are universal across people and languages.

"One of the big advances of the human brain was the ability to combine individual concepts into complex thoughts, to think not just of 'bananas,' but 'I like to eat bananas in evening with my friends,'" said Just, the D.O. Hebb University Professor of Psychology in the Dietrich College of Humanities and Social Sciences. "We have finally developed a way to see thoughts of that complexity in the fMRI signal. The discovery of this correspondence between thoughts and brain activation patterns tells us what the thoughts are built of."

Previous work by Just and his team showed that thoughts of familiar objects, like bananas or hammers, evoke activation patterns that involve the neural systems that we use to deal with those objects. For example, how you interact with a banana involves how you hold it, how you bite it and what it looks like.

The new study demonstrates that the brain's coding of 240 complex events, sentences like the shouting during the trial scenario uses an alphabet of 42 meaning components, or neurally plausible semantic features, consisting of features, like person, setting, size, social interaction and physical action. Each type of information is processed in a different brain system—which is how the brain also processes the information for objects. By measuring the activation in each brain system, the program can tell what types of thoughts are being contemplated.

For seven adult participants, the researchers used a computational model to assess how the for 239 sentences corresponded to the neurally plausible semantic features that characterized each sentence. Then the program was able to decode the features of the 240th left-out sentence. They went through leaving out each of the 240 sentences in turn, in what is called cross-validation.

The model was able to predict the features of the left-out sentence, with 87 percent accuracy, despite never being exposed to its activation before. It was also able to work in the other direction, to predict the activation pattern of a previously unseen sentence, knowing only its semantic features.

"Our method overcomes the unfortunate property of fMRI to smear together the signals emanating from brain events that occur close together in time, like the reading of two successive words in a sentence," Just said. "This advance makes it possible for the first time to decode thoughts containing several concepts. That's what most human thoughts are composed of."

He added, "A next step might be to decode the general type of topic a person is thinking about, such as geology or skateboarding. We are on the way to making a map of all the types of knowledge in the brain."

CMU's Jing Wang and Vladimir L. Cherkassky also participated in the study.

Discovering how the brain decodes complex thoughts is one of the many brain research breakthroughs to happen at Carnegie Mellon. CMU has created some of the first cognitive tutors, helped to develop the Jeopardy-winning Watson, founded a groundbreaking doctoral program in neural computation, and is the birthplace of artificial intelligence and cognitive psychology. Building on its strengths in biology, computer science, psychology, statistics and engineering, CMU launched BrainHub, an initiative that focuses on how the structure and activity of the brain give rise to complex behaviors.

Explore further: Brain 'reads' sentences the same in English and Portuguese

More information: Human Brain Mapping , www.ccbi.cmu.edu/reprints/Wang … Journal-preprint.pdf

Related Stories

Brain 'reads' sentences the same in English and Portuguese

November 3, 2016
An international research team led by Carnegie Mellon University has found that when the brain "reads" or decodes a sentence in English or Portuguese, its neural activation patterns are the same.

Scientists discover how the brain repurposes itself to learn scientific concepts

April 12, 2016
The human brain was initially used for basic survival tasks, such as staying safe and hunting and gathering. Yet, 200,000 years later, the same human brain is able to learn abstract concepts, like momentum, energy and gravity, ...

Brain representations of social thoughts accurately predict autism diagnosis

December 2, 2014
Psychiatric disorders—including autism—are characterized and diagnosed based on a clinical assessment of verbal and physical behavior. However, brain imaging and cognitive neuroscience are poised to provide a powerful ...

This is your brain on sentences

August 15, 2016
Researchers at the University of Rochester have, for the first time, decoded and predicted the brain activity patterns of word meanings within sentences, and successfully predicted what the brain patterns would be for new ...

Take a look, and you'll see, into your imagination

May 31, 2017
Scanning your brain to decode the contents of your mind has been a subject of intense research interest for some time. As studies have progressed, scientists have gradually been able to interpret what test subjects see, remember, ...

Recommended for you

Now you like it, now you don't: Brain stimulation can change how much we enjoy and value music

November 20, 2017
Enjoyment of music is considered a subjective experience; what one person finds gratifying, another may find irritating. Music theorists have long emphasized that although musical taste is relative, our enjoyment of music, ...

Deletion of a stem cell factor promotes TBI recovery in mice

November 20, 2017
UT Southwestern molecular biologists today report the unexpected finding that selectively deleting a stem cell transcription factor in adult mice promotes recovery after traumatic brain injury (TBI).

Brain cell advance brings hope for Creutzfeldt-Jakob disease

November 20, 2017
Scientists have developed a new system to study Creutzfeldt-Jakob disease in the laboratory, paving the way for research to find treatments for the fatal brain disorder.

MRI uncovers brain abnormalities in people with depression and anxiety

November 20, 2017
Researchers using MRI have discovered a common pattern of structural abnormalities in the brains of people with depression and social anxiety, according to a study presented being next week at the annual meeting of the Radiological ...

Neuroscience research provides evidence the brain is strobing, not constant

November 17, 2017
It's not just our eyes that play tricks on us, but our ears. That's the finding of a landmark Australian-Italian collaboration that provides new evidence that oscillations, or 'strobes', are a general feature of human perception.

Brain activity buffers against worsening anxiety

November 17, 2017
Boosting activity in brain areas related to thinking and problem-solving may also buffer against worsening anxiety, suggests a new study by Duke University researchers.

1 comment

Adjust slider to filter visible comments by rank

Display comments: newest first

Spaced out Engineer
not rated yet Jun 26, 2017
This technology may already exist and is being used against thousands of Americans for torture.
http://www.bigger...den.com/
There is something like 300 degrees of freedom for any given word, so the order of magnitude presented here makes sense. For other regions without clear delineation and functional partitioning, multiple realizability maybe the saving grace, from those who take a connnectionalist stance.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.