Picture an air-traffic controller tracking 10 planes approaching an airport. Now imagine he's having trouble focusing on all 10 aircraft, perhaps because he's been up all night or just has a lot on his mind. What would happen if his computer sensed his mental fatigue, removed one plane from his oversight and reassigned it to a controller who just started her shift?
The scenario might seem like science fiction, but with new technology being developed by Tufts researchers Robert Jacob and Sergio Fantini, it could be quite real someday. Jacob and Fantini have developed a brain-scanning device that allows a computer to sense the level of mental exertion of its user and adjust tasks accordingly to achieve the correct balance between boredom and overload.
"Humans and computers are two powerful information processors connected by this miserably narrow bandwidth—a mouse and a keyboard," says Jacob, a professor of computer science in the School of Engineering. Jacob's challenge is to find ways to create a more direct connection between machine and human brain to make both more efficient.
Eight years ago, he walked into the lab of Fantini, a professor of biomedical engineering at Tufts, and saw him experimenting with a technique called diffuse optical spectroscopy, which uses light to peer inside the body. Initially employed in the 1920s to examine cancerous tissue, it more recently has been used to look at brain function. "We illuminate the tissue, in this case brain tissue, with a laser diode," says Fantini. "Because the light diffuses, some of it will travel to the brain cortex and make it back."
In the video above, Rob Jacob and Sergio Fantini talk about their work on the mind-reading computer. Video: Steffan Hacker
The amount of light that is absorbed depends on how much oxygen is circulating through the arteries and veins bringing blood to and from the brain. When a particular part of the brain works hard, it consumes oxygen more rapidly, sending a signal to the body to dilate the arteries and bring more oxygen-rich blood to that area. At the same time, the light that returns to the scanner is red or infrared in color, since that is the only wavelength that can penetrate the skull. By measuring the change in color of the diffused light, researchers can get a sense of how much mental effort a person is applying to a task.
"When I found out Sergio had a new technique for measuring what is happening in your brain, I said, yes! We can use this as an additional channel of communicating information" between humans and computers, says Jacob. While many researchers studying brain-computer interaction focus on creating devices to help disabled people, Jacob and Fantini want to develop new interfaces for the average user.
It's an important distinction. In the case of disabled people, researchers work with direct control, in which users consciously think of a motion or command in order to perform a particular task. Jacob and Fantini, however, are working with passive control, in which a computer senses incidental brain activity and automatically adjusts its function, like lightening the load of the exhausted air traffic controller. No explicit user action required.
To achieve that passive control, the two researchers created a device—a sports headband that holds a collection of tiny fiber optic cables tight against a user's forehead—that sends beams of near-infrared light through the skull and about three centimeters into the brain. Fantini compares it to the way your fingers take on a reddish glow when you hold a flashlight to them. (For a later iteration of the sensing device, Jacob and Fantini collaborated with costume designers in Tufts' drama department to design a custom headband.)
It's a far cry from other brain-scanning techniques, such as functional magnetic resonance imaging (fMRI), which requires users to lie perfectly still inside a giant machine. With the headband scanner, you can sit and use a computer in an ordinary office setting.
Jacob hastens to add that the device does not read the content of thoughts—only the level of brain activity. "Of course we might be able to measure that in the future, who knows," he adds. "Then another argument comes into play, which is the question of whether it's your personal device and it's helping you, or whether your employer is using it to monitor you."
Such questions are far in the future, however. At this point, the researchers are conceiving of the device only as one that will help users maximize their brain energy and productivity.
To test their invention, Jacob and Fantini initially experimented with simple memory tasks, measuring the increase in blood flow to the part of the cerebral cortex dealing with attention and short-term memory. That turned out to be quite difficult. Blood flow to the brain changes with every motion of the body—and even fluctuates when you're sitting still. "Even when you sleep, the brain is not idle, so it's never just a flat signal," says Fantini.
In order to account for this, the researchers let the computer program do the work—relying on a technique called "machine learning," in which the computer examines incoming data and makes new calculations based on past experience. "We start with known examples of high and low workload, and have it come up with an answer," says Jacob.
After calibrating the device, they developed several simulations that allowed the computer to directly communicate with the brain, and set up studies along with graduate students to test them. The first was an airplane simulation similar to the air-traffic controller juggling too many incoming flights. Study participants wearing the headband were told to guide several planes to their proper destinations. When the computer recognized that a person's attention was slipping, it would surreptitiously remove one of the planes. When it saw that a person was getting bored, it would add an airplane to keep him or her engaged.
By optimizing the number of airplanes, the researchers found that they could improve participants' performance in correctly guiding planes to their destinations. "It's the kind of thing that if we had a knob that said, 'Now I'm feeling busy; now I'm feeling not so busy,' we might do ourselves, but that would be very distracting," says Jacob. It would also be less effective.
The brain scanner makes the computer a better judge of when we are getting tired or distracted. In fact, Jacob and Fantini worked hard to make the changes in their simulation so subtle that participants hardly noticed when a plane appeared or disappeared on the screen.
They are developing additional applications for similar situations in which a person might have varying levels of attention for a task. One is a special cursor that grows larger or smaller in order to select items as you use a mouse to drag it across a screen. By measuring the attention span of the user, it can better anticipate what item he or she is trying to select.
Prioritizing Our Work
"When we are overworked, we overemphasize the high-priority topics, making the low-priority topics harder to get to," says Jacob. The same principle, he says, could be adapted to web browsing or word processing, to give you more information when you have more mental capacity to process it, and less when your brain needs a rest.
The researchers have also been working with Google to create a more streamlined version of the device. In the future, it could possibly be integrated into Google Glass—a small wearable computer that fits onto a pair of eyeglasses—in order to tweak the screen display that viewers see, giving them more or less information depending on their brain activity.
Currently, the monitoring unit that the headband is hooked up to is the size of a stereo receiver. It would require some significant engineering to shrink the device to a size that would make it truly transportable, but the researchers say it is possible. "The brain scanner is just a little light and a sensor," says Jacob. "It's easy to imagine a relatively cheap device that we can stick up against your forehead." Graduate student Chirag Sthalekar is working on a new interface for the device in the form of a circuit board only a couple of inches on each side.
At the same time, the current device is limited to measuring brain activity only through skin on the forehead. The lab is developing a sensor that could work around hair more easily to measure parts of the brain controlling other functions and experimenting with other body sensors to measure attributes such as heart rate and galvanic skin response (small electrical charges in the skin due to stress or emotion). One day, our computers may know more about our state of mind than we do—and help us be more productive.