Do you see what I see? Researchers harness brain waves to reconstruct images of what we perceive

February 22, 2018 by Don Campbell, University of Toronto
Dan Nemrodov (left) and Professor Adrian Nestor (left) have developed a technique that can harness brain waves gathered by data to show how our brains perceive images of faces . Credit: Ken Jones

A new technique developed by neuroscientists at the University of Toronto Scarborough can, for the first time, reconstruct images of what people perceive based on their brain activity gathered by EEG.

The technique developed by Dan Nemrodov, a postdoctoral fellow in Assistant Professor Adrian Nestor's lab at U of T Scarborough, is able to digitally reconstruct images seen by test subjects based on electroencephalography (EEG) data.

"When we see something, our creates a mental percept, which is essentially a mental impression of that thing. We were able to capture this percept using EEG to get a direct illustration of what's happening in the brain during this process," says Nemrodov.

For the study, hooked up to EEG equipment were shown images of faces. Their was recorded and then used to digitally recreate the image in the subject's mind using a technique based on machine learning algorithms.

It's not the first time researchers have been able to reconstruct images based on visual stimuli using neuroimaging techniques. The current method was pioneered by Nestor who successfully reconstructed facial images from imaging (fMRI) data in the past, but this is the first time EEG has been used.

And while techniques like fMRI - which measures brain activity by detecting changes in blood flow - can grab finer details of what's going on in specific areas of the brain, EEG has greater practical potential given that it's more common, portable, and inexpensive by comparison. EEG also has greater temporal resolution, meaning it can measure with detail how a percept develops in time right down to milliseconds, explains Nemrodov.

"fMRI captures activity at the time scale of seconds, but EEG captures activity at the millisecond scale. So we can see with very fine detail how the percept of a face develops in our brain using EEG," he says. In fact, the researchers were able to estimate that it takes our brain about 170 milliseconds (0.17 seconds) to form a good representation of a face we see.

This study provides validation that EEG has potential for this type of notes Nemrodov, something many researchers doubted was possible given its apparent limitations. Using EEG data for image reconstruction has great theoretical and practical potential from a neurotechnological standpoint, especially since it's relatively inexpensive and portable.

In terms of next steps, work is currently underway in Nestor's lab to test how image reconstruction based on EEG data could be done using memory and applied to a wider range of objects beyond faces. But it could eventually have wide-ranging clinical applications as well.

"It could provide a means of communication for people who are unable to verbally communicate. Not only could it produce a neural-based reconstruction of what a person is perceiving, but also of what they remember and imagine, of what they want to express," says Nestor.

"It could also have forensic uses for law enforcement in gathering eyewitness information on potential suspects rather than relying on verbal descriptions provided to a sketch artist."

The research, which will be published in the journal eNeuro, was funded by the Natural Sciences and Engineering Research Council of Canada (NSERC) and by a Connaught New Researcher Award.

"What's really exciting is that we're not reconstructing squares and triangles but actual images of a person's face, and that involves a lot of fine-grained visual detail," adds Nestor.

"The fact we can reconstruct what someone experiences visually based on their brain activity opens up a lot of possibilities. It unveils the subjective content of our mind and it provides a way to access, explore and share the content of our perception, memory and imagination."

Explore further: Researchers in Japan are showing way to decode thoughts

More information: Dan Nemrodov et al. The Neural Dynamics of Facial Identity Processing: insights from EEG-Based Pattern Analysis and Image Reconstruction, eneuro (2018). DOI: 10.1523/ENEURO.0358-17.2018

Related Stories

Researchers in Japan are showing way to decode thoughts

January 15, 2018
Making news this month is a study by researchers the Advanced Telecommunications Research Institute International (ATR) and Kyoto University in Japan, having built a neural network that not only reads but re-creates what ...

Can't get an image out of your head? Your eyes are helping to keep it there

February 14, 2018
Even though you are not aware of it, your eyes play a role in searing an image into your brain, long after you have stopped looking at it.

Researchers explore decoding faces from neural activity

June 15, 2016
(Tech Xplore)—What can technology accomplish in reconstructing a face? Scientists have come up with surprising results. Yale Daily News in 2014 reported on a study about how, with a brain scan, a face is reconstructed—neuroscience ...

Researchers demonstrate 'mind-reading' brain-decoding tech

October 23, 2017
Researchers have demonstrated how to decode what the human brain is seeing by using artificial intelligence to interpret fMRI scans from people watching videos, representing a sort of mind-reading technology.

Using deep learning to read images being processed in the brain

May 8, 2017
(Tech Xplore)—A team of researchers from several institutions in China has applied deep learning by a computer to the problem of reading visual imagery in the brain and then reproduced it in a 2-D format. A paper describing ...

Take a look, and you'll see, into your imagination

May 31, 2017
Scanning your brain to decode the contents of your mind has been a subject of intense research interest for some time. As studies have progressed, scientists have gradually been able to interpret what test subjects see, remember, ...

Recommended for you

Neuroimaging study reveals 'hot spot' for cue-reactivity in substance-dependent population

November 20, 2018
When patients with dependence on alcohol, cocaine or nicotine are shown drug cues, or images related to the substance, an area of their brain known as the medial prefrontal cortex (mPFC) shows increased activity, report investigators ...

When storing memories, brain prioritizes those experiences that are most rewarding

November 20, 2018
The brain's ability to preserve memories lies at the heart of our basic human experience. But how does the brain's mechanism for memory make sure we remember the most significant events and not clog our minds with superfluous ...

To predict the future, the brain has two clocks

November 20, 2018
That moment when you step on the gas pedal a split second before the light changes, or when you tap your toes even before the first piano note of Camila Cabello's "Havana" is struck. That's anticipatory timing.

Researchers hope to be able to replace dysfunctional brain cells

November 20, 2018
A new study by researchers at Karolinska Institutet supports the theory that replacement of dysfunctional immune cells in the brain has therapeutic potential for neurodegenerative diseases like ALS and Alzheimer's disease. ...

White matter pathway and individual variability in human stereoacuity

November 20, 2018
Researchers in the Center for Information and Neural Networks (CiNet), the National Institute of Information and Communications Technology and Osaka University have identified a human white matter pathway associated with ...

Can genetic therapy help kids with Angelman syndrome overcome seizures?

November 20, 2018
Angelman syndrome is a genetic disease with no cure. Children grow up with severe intellectual disabilities and a range of other problems, arguably the worst of which are epileptic seizures. Now scientists at the UNC School ...

1 comment

Adjust slider to filter visible comments by rank

Display comments: newest first

flashgordon
not rated yet Feb 22, 2018
wait till they get a load of me! ah . . . he, he, he!

https://www.youtu...FXnj9Nz8

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.