Brain imaging reveals the movies in our mind

September 22, 2011 in Neuroscience
Brain scans let computer reconstruct movie scenes (AP)

Enlarge

This set of paired images provided by Shinji Nishimoto of the University of California, Berkeley on Wednesday, Sept. 21, 2011 shows original video images, upper row, and those images reconstructed by computer from brain scans. While volunteers watched movie clips, a scanner watched their brains. And from their brain activity, a computer made rough reconstructions of what they viewed. Scientists reported that result Thursday, Sept. 22, 2011 and speculated such an approach might be able to reveal dreams and hallucinations someday. In the future, it might help stroke victims or others who have no other way to communicate, said Jack Gallant, a neuroscientist at the University of California, Berkeley, and co-author of the paper. (University of California, Berkeley, Shinji Nishimoto)

Imagine tapping into the mind of a coma patient, or watching one's own dream on YouTube. With a cutting-edge blend of brain imaging and computer simulation, scientists at the University of California, Berkeley, are bringing these futuristic scenarios within reach.

Using () and computational models, UC Berkeley researchers have succeeded in decoding and reconstructing people's dynamic visual experiences – in this case, watching Hollywood movie trailers.

As yet, the technology can only reconstruct movie clips people have already viewed. However, the breakthrough paves the way for reproducing the movies inside our heads that no one else sees, such as dreams and memories, according to researchers.

"This is a major leap toward reconstructing internal imagery," said Professor Jack Gallant, a UC Berkeley neuroscientist and coauthor of the study to be published online Sept. 22 in the journal Current Biology. "We are opening a window into the movies in our minds."

Eventually, practical applications of the technology could include a better understanding of what goes on in the minds of people who cannot communicate verbally, such as stroke victims, coma patients and people with neurodegenerative diseases.

It may also lay the groundwork for brain-machine interface so that people with cerebral palsy or paralysis, for example, can guide computers with their minds.

However, researchers point out that the technology is decades from allowing users to read others' thoughts and intentions, as portrayed in such sci-fi classics as "Brainstorm," in which scientists recorded a person's sensations so that others could experience them.

Previously, Gallant and fellow researchers recorded brain activity in the visual cortex while a subject viewed black-and-white photographs. They then built a that enabled them to predict with overwhelming accuracy which picture the subject was looking at.

In their latest experiment, researchers say they have solved a much more difficult problem by actually decoding brain signals generated by moving pictures.

"Our natural visual experience is like watching a movie," said Shinji Nishimoto, lead author of the study and a post-doctoral researcher in Gallant's lab. "In order for this technology to have wide applicability, we must understand how the brain processes these dynamic visual experiences."

Nishimoto and two other research team members served as subjects for the experiment, because the procedure requires volunteers to remain still inside the MRI scanner for hours at a time.

They watched two separate sets of Hollywood movie trailers, while fMRI was used to measure blood flow through the visual cortex, the part of the brain that processes visual information. On the computer, the brain was divided into small, three-dimensional cubes known as volumetric pixels, or "voxels."

"We built a model for each voxel that describes how shape and motion information in the movie is mapped into brain activity," Nishimoto said.

The brain activity recorded while subjects viewed the first set of clips was fed into a computer program that learned, second by second, to associate visual patterns in the movie with the corresponding brain activity.

Brain activity evoked by the second set of clips was used to test the movie reconstruction algorithm. This was done by feeding 18 million seconds of random YouTube videos into the computer program so that it could predict the brain activity that each film clip would most likely evoke in each subject.

Finally, the 100 clips that the computer program decided were most similar to the clip that the subject had probably seen were merged to produce a blurry yet continuous reconstruction of the original movie.

Reconstructing movies using brain scans has been challenging because the blood flow signals measured using fMRI change much more slowly than the neural signals that encode dynamic information in movies, researchers said. For this reason, most previous attempts to decode have focused on static images.

"We addressed this problem by developing a two-stage model that separately describes the underlying neural population and blood flow signals," Nishimoto said.

Ultimately, Nishimoto said, scientists need to understand how the brain processes dynamic visual events that we experience in everyday life.

"We need to know how the brain works in naturalistic conditions," he said. "For that, we need to first understand how the works while we are watching movies."

More information:
Current Biology: http://www.cell.com/current-biology

Movie reconstructions: http://bit.ly/nzm9Tw

Provided by University of California - Berkeley search and more info website

3.9 /5 (35 votes)  

Filter


Move the slider to adjust rank threshold, so that you can hide some of the comments.


Display comments: newest first

shwhjw
Sep 22, 2011

Rank: 5 / 5 (1)
Another application for being able to reconstruct a person's mental image is art: you look at a blank screen and imagine a picture, a blurry shape comes out. You visualise detail on this blurry shape, and it appears. Repeat for more and more detail!
krundoloss
Sep 22, 2011

Rank: 5 / 5 (1)
Thats great, just dont let my wife see the images in my head of all her hot friends........
Isaacsname
Sep 22, 2011

Rank: not rated yet
.....Ruh-roh.

I just see it now. In the future we'll have all sorts of people plugged in, vacantly staring at the sky like turkeys, the meaning of the word " interweb " will take on a whole new meaning.
Birthmark
Sep 22, 2011

Rank: not rated yet
The pictures formed from the brain are beautiful. This will be very interesting when this technique could be mastered in the future; it has the potential for a lot of implications.
antonima
Sep 22, 2011

Rank: not rated yet
I don't know, but from the pictures it seems that the machine is reading the person's subconscious view rather than the conscious image. The actor is seen just like an ordinary guy; some people will try to see movies through an actor's eyes.
The plane looks like a building on the horizon; perhaps the brain just subcategorizes images into more manageable paradigms.
When someone sees an elephant, they just think - hey its an 'elephant'. They rarely have actual experience with these creatures, instead they think they know, or rather, they can identify what it is and it suffices for their purposes. Thats why the elephant is just a blob! The background is lush - blue and green. In most people's eyes elephants are exotic creatures in exotic locales.

It would be very very interesting to see the same clips through different people's eyes!
Jeddy_Mctedder
Sep 23, 2011

Rank: 1 / 5 (2)
i call shenanigans on this. they are promising much more than they will ever be able to deliver. remote sensing (fmri) will NEVER be able to do anything like this . it is total bullshit. wake me up when there's some sort of set of implanted multi-pin electrodes and maybe i'll believe this crap. yea, 'simulating' what a person is seeing, when people themeselves frequently don't know what they are consciously seeing.
Bob_Kob
Sep 23, 2011

Rank: not rated yet
@antonima

Thats not the case. They said the reason why the elephant didnt come out well was because none of the youtube videos used to reconstruct the images contained elephants!
krundoloss
Sep 23, 2011

Rank: 5 / 5 (1)
Thats really cool! Its amazing! We are "analog" machines, that sort of output looks pretty good! The thing about our "inner eye" that makes it harder to reproduce is that we only create the images we require, no extra detail, just enough.
Rank 3.9 /5 (35 votes)
Related Stories
Relevant PhysicsForums posts
  • A question about drug tolerance
    createdMay 23, 2012
  • Poor nutrition leading to overeating?
    createdMay 23, 2012
  • Math and dyslexia?
    createdMay 21, 2012
  • portable metabolism meter?
    createdMay 21, 2012
  • Rare medical conditions on 20/20 tonight
    createdMay 18, 2012
  • "Good" Cholesterol in Doubt
    createdMay 17, 2012
  • More from Physics Forums - Medical Sciences

More news stories

Of mice and mental models: Neuroscientific implications of risk-optimized behavior in the mouse

(Medical Xpress) -- Regardless of an organism’s biological complexity, every encephalized animal continuously makes under-informed behavioral choices that can have serious consequences. Despite its ubiquity, ...

Neuroscience created 18 hours ago | popularity not rated yet | comments 0 | with audio podcast feature

Persistent sensory experience is good for aging brain

Despite a long-held scientific belief that much of the wiring of the brain is fixed by the time of adolescence, a new study shows that changes in sensory experience can cause massive rewiring of the brain, even as one ages. ...

Neuroscience created May 24, 2012 | popularity not rated yet | comments 0

Boundary stops molecule right where it needs to be

A molecule responsible for the proper formation of a key portion of the nervous system finds its way to the proper place not because it is actively recruited, but instead because it can't go anywhere else.

Neuroscience created May 24, 2012 | popularity not rated yet | comments 0 | with audio podcast

Locating ground zero: How the brain's emergency workers find the disaster area

Like emergency workers rushing to a disaster scene, cells called microglia speed to places where the brain has been injured, to contain the damage by 'eating up' any cellular debris and dead or dying neurons. ...

Neuroscience created May 24, 2012 | popularity 5 / 5 (2) | comments 0 | with audio podcast

Genetic 'reset switch' enables signaling pathway to induce multiple developmental outcomes for olfactory neurons

Within the nervous system, a handful of signaling pathways modulate development of a cornucopia of different neuronal subtypes. “Even small alterations in neuron differentiation pathways can disrupt subsequent ...

Neuroscience created May 24, 2012 | popularity not rated yet | comments 0


Tongue analysis software uses ancient Chinese medicine to warn of disease

For 5,000 years, the Chinese have used a system of medicine based on the flow and balance of positive and negative energies in the body. In this system, the appearance of the tongue is one of the measures used to classify ...

Cancer may require simpler genetic mutations than previously thought

Chromosomal deletions in DNA often involve just one of two gene copies inherited from either parent. But scientists haven't known how a deletion in one gene from one parent, called a "hemizygous" deletion, can contribute ...

Inherited DNA change explains overactive leukemia gene

A small inherited change in DNA is largely responsible for overactivating a gene linked to poor treatment response in people with acute leukemia.

Skp2 activates cancer-promoting, glucose-processing Akt

HER2 and its epidermal growth factor receptor cousins mobilize a specialized protein to activate a major player in cancer development and sugar metabolism, scientists report in the May 25 issue of Cell.

Early physical therapist treatment associated with reduced risk of healthcare utilization and reduced overall healthcare

A new study published in Spine shows that early treatment by a physical therapist for low back pain (LBP), as compared to delayed treatment, was associated with reduced risk of subsequent healthcare utilization and lower ...

New device allows pacemaker patients to safely undergo MRIs

For many, it's a medical conundrum: The very pacemaker keeping their heart in rhythm prevents them from undergoing an MRI to diagnose other ailments, because interaction between the two devices could prove deadly.