Princeton-Intel collaboration breaks new ground in studies of the brain

February 24, 2017 by Catherine Zandonella, Princeton University
Credit: public domain

Early this year, about 30 neuroscientists and computer programmers got together to improve their ability to read the human mind.

The hackathon was one of several that researchers from Princeton University and Intel, the largest maker of computer processors, organized to build software that can tell what a person is thinking in , while the person is thinking it.

The between researchers at Princeton and Intel has enabled rapid progress on the ability to decode digital data, scanned using functional magnetic resonance imaging (fMRI), to reveal how neural activity gives rise to learning, memory and other cognitive functions.

A review of computational advances toward decoding appears in the journal Nature Neuroscience, authored by researchers at the Princeton Neuroscience Institute and Princeton's departments of computer science and electrical engineering, together with colleagues at Intel Labs, a research arm of Intel.

"The capacity to monitor the brain in real time has tremendous potential for improving the diagnosis and treatment of brain disorders as well as for basic research on how the mind works," said Jonathan Cohen, the Robert Bendheim and Lynn Bendheim Thoman Professor in Neuroscience, co-director of the Princeton Neuroscience Institute, and one of the founding members of the collaboration with Intel.

Since the collaboration's inception two years ago, the researchers have whittled the time it takes to extract thoughts from brain scans from days down to less than a second, said Cohen, who is also a professor of psychology.

One type of experiment that is benefiting from real-time decoding of thoughts occurred during the hackathon. The study, designed by J. Benjamin Hutchinson, a former postdoctoral researcher in the Princeton Neuroscience Institute who is now an assistant professor at Northeastern University, aimed to explore activity in the brain when a person is paying attention to the environment, versus when his or her attention wanders to other thoughts or memories.

In the experiment, Hutchinson asked a research volunteer—a graduate student lying in the fMRI scanner—to look at a detail-filled picture of people in a crowded café. From his computer in the console room, Hutchinson could tell in real time whether the graduate student was paying attention to the picture or whether her mind was drifting to internal thoughts. Hutchinson could then give the feedback on how well she was paying attention by making the picture clearer and stronger in color when her mind was focused on the picture, and fading the picture when her attention drifted.

The ongoing collaboration has benefited neuroscientists who want to learn more about the brain and computer scientists who want to design more efficient computer algorithms and processing methods to rapidly sort through large data sets, according to Theodore Willke, a senior principal engineer at Intel Labs in Hillsboro, Oregon, and head of Intel's Mind's Eye Lab. Willke directs Intel's part of the collaborative team.

"Intel was interested in working on emerging applications for high-performance computing, and the collaboration with Princeton provided us with new challenges," Willke said. "We also hope to export what we learn from studies of human intelligence and cognition to machine learning and artificial intelligence, with the goal of advancing other important objectives, such as safer autonomous driving, quicker drug discovery and ealier detection of cancer."

Since the invention of fMRI two decades ago, researchers have been improving the ability to sift through the enormous amounts of data in each scan. An fMRI scanner captures signals from changes in blood flow that happen in the brain from moment to moment as we are thinking. But reading from these measurements the actual thoughts a person is having is a challenge, and doing it in real time is even more challenging.

A number of techniques for processing these data have been developed at Princeton and other institutions. For example, work by Peter Ramadge, the Gordon Y.S. Wu Professor of Engineering and professor of electrical engineering at Princeton, has enabled researchers to identify brain activity patterns that correlate to thoughts by combining data from brain scans from multiple people. Designing computerized instructions, or algorithms, to carry out these analyses continues to be a major area of research.

Powerful high-performance computers help cut down the time that it takes to do these analyses by breaking the task up into chunks that can be processed in parallel. The combination of better algorithms and parallel computing is what enabled the collaboration to achieve real-time brain scan processing, according to Kai Li, Princeton's Paul M. Wythes '55 P86 and Marcia R. Wythes P86 Professor in Computer Science and one of the founders of the collaboration.

Since the beginning of the collaboration in 2015, Intel has contributed to Princeton more than $1.5 million in computer hardware and support for Princeton graduate students and postdoctoral researchers. Intel also employs 10 computer scientists who work on this project with Princeton, and these experts work closely with Princeton faculty, students and postdocs to improve the software.

These algorithms locate thoughts within the data by using machine learning, the same technique that facial recognition software uses to help find friends in social media platforms such as Facebook. Machine learning involves exposing computers to enough examples so that the computers can classify new objects that they've never seen before.

One of the results of the collaboration has been the creation of a software toolbox, called the Brain Imaging Analysis Kit (BrainIAK), that is openly available via the Internet to any researchers looking to process fMRI data. The team is now working on building a real-time analysis service. "The idea is that even researchers who don't have access to high-performance computers, or who don't know how to write software to run their analyses on these computers, would be able to use these tools to decode brain scans in real time," said Li.

What these scientists learn about the brain may eventually help individuals combat difficulties with paying attention, or other conditions that benefit from immediate feedback.

For example, real-time feedback may help patients train their brains to weaken intrusive memories. While such "brain-training" approaches need additional validation to make sure that the brain is learning new patterns and not just becoming good at doing the training exercise, these feedback approaches offer the potential for new therapies, Cohen said. Real-time analysis of the brain could also help clinicians make diagnoses, he said.

The ability to decode the brain in real time also has applications in basic brain research, said Kenneth Norman, professor of psychology and the Princeton Neuroscience Institute. "As cognitive neuroscientists, we're interested in learning how the brain gives rise to thinking," said Norman. "Being able to do this in real time vastly increases the range of science that we can do," he said.

Another way the technology can be used is in studies of how we learn. For example, when a person listens to a math lecture, certain neural patterns are activated. Researchers could look at the neural patterns of people who understand the math lecture and see how they differ from neural patterns of someone who isn't following along as well, according to Norman.

The ongoing collaboration is now focused on improving the technology to obtain a clearer window into what people are thinking about, for example, decoding in real time the specific identity of a face that a person is mentally visualizing.

One of the challenges the computer scientists had to overcome was how to apply machine learning to the type of data generated by brain scans. A face-recognition algorithm can scan hundreds of thousands of photographs to learn how to classify new faces, but the logistics of scanning peoples' brains are such that researchers usually only have access to a few hundred scans per person.

Although the number of scans is few, each scan contains a rich trove of data. The software divides the brain images into little cubes, each about one millimeter wide. These cubes, called voxels, are analogous to the pixels in a two-dimensional picture. The brain activity in each cube is constantly changing.

To make matters more complex, it is the connections between brain regions that give rise to our thoughts. A typical scan can contain 100,000 voxels, and if each voxel can talk to all the other voxels, the number of possible conversations is immense. And these conversations are changing second by second. The collaboration of Intel and Princeton computer scientists overcame this computational challenge. The effort included Li as well as Barbara Engelhardt, assistant professor of computer science, and Yida Wang, who earned his doctorate in computer science from Princeton in 2016 and now works at Intel Labs.

Prior to the recent progress, it would take months to analyze a data set, said Nicholas Turk-Browne, professor of psychology at Princeton. With the availability of real-time fMRI, a researcher can change the experiment while it is ongoing. "If my hypothesis concerns a certain region of the brain and I detect in real time that my experiment is not engaging that brain region, then we can change what we ask the research volunteer to do to better engage that region, potentially saving precious time and accelerating scientific discovery," Turk-Browne said.

One eventual goal is to be able to create pictures from people's thoughts, said Turk-Browne. "If you are in the scanner and you are retrieving a special memory, such as from childhood, we would hope to generate a photograph of that experience on the screen. That is still far off, but we are making good progress."

The paper, "Computational approaches to fMRI analysis," by Jonathan Cohen, Nathaniel Daw, Barbara Engelhardt, Uri Hasson, Kai Li, Yael Niv, Kenneth Norman, Jonathan Pillow, Peter Ramadge, Nicholas Turk-Browne and Theodore Willke, appeared in the March 2017 issue of the journal Nature Neuroscience.

Explore further: Brain scanners allow scientists to 'read minds'—could they now enable a 'Big Brother' future?

More information: Jonathan D Cohen et al, Computational approaches to fMRI analysis, Nature Neuroscience (2017). DOI: 10.1038/nn.4499

Related Stories

Brain scanners allow scientists to 'read minds'—could they now enable a 'Big Brother' future?

February 9, 2017
Are you lying? Do you have a racial bias? Is your moral compass intact?To find out what you think or feel, we usually have to take your word for it. But questionnaires and other explicit measures to reveal what's on your ...

How memories shape our perception of the present

January 6, 2017
What are memories made of? Do different parts of our brain light up when we perceive an event than when we remember it afterward? What role does memory play in directing our attention to specific details in our surroundings?

Real-time brain feedback reduces attention lapses

February 11, 2015
Your attention, please. People make mistakes every day because they lose focus. Maybe your car drifts across the center line or an error slips into a report at work.

'Pay no attention to that man behind the curtain': researchers show how human brain decides what's important

January 18, 2017
The Wizard of Oz told Dorothy to "pay no attention to that man behind the curtain" in an effort to distract her, but a new Princeton University study sheds light on how people learn and make decisions in real-world situations.

Study uses 'Sherlock' to reveal how shared experiences shape our memories

December 6, 2016
We tend to think of our memories as unique, but a Princeton University-led study shows that memories are often shared rather than idiosyncratic.

Missed connections: As people age, memory-related brain activity loses cohesion

November 23, 2016
Groups of brain regions that synchronize their activity during memory tasks become smaller and more numerous as people age, according to a study published in PLOS Computational Biology.

Recommended for you

New link between sleep arousals and body temperature may also be connected to SIDS

April 25, 2018
Brief arousals during sleep—sometimes as many as ten to fifteen per night—appear random in time and occur in humans and even in animals.

Ethics debate overdue in human brain research: experts

April 25, 2018
What if human brain tissue implanted into a pig transferred some of the donor's self-awareness and memories?

Imaging may allow safe tPA treatment of patients with unwitnessed strokes

April 25, 2018
A study led by Massachusetts General Hospital (MGH) investigators may lead to a significant expansion in the number of stroke patients who can safely be treated with intravenous tPA (tissue plasminogen activator), the "clot ...

Brain structure linked to symptoms of restless legs syndrome

April 25, 2018
People with restless legs syndrome may have changes in a portion of the brain that processes sensory information, according to a study published in the April 25, 2018, online issue of Neurology, the medical journal of the ...

Brain activity linked to stress changes chemical codes

April 24, 2018
Five years ago, a team of University of California San Diego neurobiologists published surprising findings describing how rats' brain cells adopted new chemical codes when subjected to significant changes in natural light ...

Scientists develop new method that uses light to manage neuropathic pain in mice

April 24, 2018
For patients with neuropathic pain, a chronic condition affecting 7 to 8 percent of the European population, extreme pain and sensitivity are a daily reality. There is currently no effective treatment. Scientists from EMBL ...


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.