Helping computers see like people
UA cognitive scientist Mary Peterson, who studies human vision, will work with collaborators from four partner institutions, funded by an Office of Naval Research grant.
If you've ever posted a photo on Facebook of yourself and a group of friends, you've probably seen a prompt asking if you'd like to tag yourself or specific friends in the image.
Using facial recognition technology, Facebook is able to identify who is in the photo and suggest which of your friends appear.
This is just one example of how computers "see." And while the science is still imperfect, researchers hope that, one day, computers may be able to see as well as humans.
University of Arizona cognitive scientist Mary Peterson is one researcher who is working to improve computer vision as part of a national, multidisciplinary team.
Peterson, who studies human vision, is a professor and director of the UA's Cognitive Science Program and also a professor of psychology in the College of Science's School of Mind, Brain and Behavior.
She and five collaborators from four partner institutions—Stanford University, the University of Illinois, the Massachusetts Institute of Technology and the University of California, Berkeley—have been awarded a three-year, $4.5 million grant from the Office of Naval Research for a project designed to improve computer vision. At the end of the three years, the project may be eligible for a two-year, $3 million extension.
The grant was awarded through the Office of Naval Research's Multi University Research Initiative, which aims to bring together scientists across institutions and disciplines to collaborate on scientific challenges.
Led by principal investigator and computer vision scientist Fei-Fei Li at Stanford, the research team includes two computer vision scientists and four researchers who study mammalian vision.
Peterson's portion of the research, funded at $750,000, will focus on better understanding the role of feedback connections in the way humans see, with hopes that those findings may be able to help improve computer vision.
"Computer vision scientists, for quite a number of years now, have been having competitions every year, and their goal is to have computer vision be as good as human vision," Peterson said. "Every year they come closer to that goal."
But there is still work to be done. While computers do well at reading things like faces, QR codes or satellite images, they struggle with deciphering very complex images or crowded scenes, Peterson explained. For example, if a person's face is partially obstructed by an object, like a wall, the computer may not be able to tell where the face ends and the wall begins.
"The idea is that if we understand more about how human vision does it, and in particular how human vision uses feedback connections, then one might be able to improve computer vision," Peterson said.
In human vision, when light enters the eye, it is absorbed by receptors on the back of the retina and then it's translated into neurosignals that follow a pathway from the back of the eye up into higher levels of the brain. For many years, scientists thought that conscious perception occurred when the signals hit a certain place in that pathway, making visual perception something of a "one-way street"—the result of a feed-forward connection, Peterson said.
Scientists later discovered that wherever a feed-forward connection occurs in the brain, there is also a feedback connection, with signals traveling in the other direction. There is some evidence that those feedback connections also may be critical to the vision process, yet it is unclear what their exact role is. That's what Peterson hopes to find out.
Peterson will work with undergraduate, graduate and postdoctoral students in her UA lab to conduct computer-based behavioral tests and functional magnetic resonance imaging with study participants. She also will work with her collaborator at the University of Illinois to use transcranial magnetic stimulation to better understand what happens in the brain during visual processing.
"What we're going to be testing in our studies are those places in particular where computer vision has a problem and/or those places where we think feedback connections might be really critically important," she said. "We're going to try to get really good evidence and then we'll be able to say to the computer vision scientists: 'Here's what the feedback seems to be doing, and if you implement this in your models we predict that you will come much closer to human vision.'"
Peterson, who plans to meet with collaborators several times throughout the project, said she is excited about the multidisciplinary nature of the research.
"By pulling together people who work on an issue or who are concerned about an issue but are not at the same institution, you can pull together different expertise, different methods, different approaches and maybe even different starting assumptions for a really wonderful multidisciplinary project," she said. "We're going to be challenging each other as we work together, and we'll ask the hard questions of each other and push each other to do even more research."