Expert panels successfully rate medical research proposals, big-data analysis shows
The National Institutes of Health (NIH) is easily the world's largest funder of medical research, and outside scientists perform most of the research. Panels of these investigators also select the projects that the NIH supports. With the NIH budget slowly dropping, some experts have questioned whether this "peer review" process is prone to favoritism or to avoiding risky but potentially high-payoff studies.
Boston University (BU) and Harvard economists, however, have published an extensive analysis of NIH grants in the journal Science that shows a high correlation between how projects are rated by peer review and the quality of the resulting research.
"Peer review is tremendously important for determining what research is done in the United States, and the world, and yet we don't know very much about how effective these systems are," says Leila Agha, Assistant Professor of Markets, Public Policy and Law at the Boston University Questrom School of Business and co-author of the Science paper. "Our findings suggest that the process successfully identifies research proposals that are most likely to result in high numbers of publications and citations."
"Our work shows that peer review generates insights about the potential of research proposals that can't be predicted from past publications, grant histories, or other quantitative metrics," adds Danielle Li, Assistant Professor of Business Administration at Harvard Business School and the other co-author.
Agha and Li examined more than 130,000 research projects funded by the NIH from 1980 to 2008. They looked at how each grant application was rated and its follow-up accomplishments as indicated by the related scientific papers published, the number of times those papers were cited by other publications, and the patents that were awarded based on the work.
They discovered that applications with better peer-review scores are consistently associated with better research outcomes. That finding held true even when they took into account the field of research, the year the grant was awarded, the lead investigator's previous history of scientific papers and NIH grants, and other factors.
As NIH funding has become more competitive over recent years, and the percentage of funded proposals has fallen, there has been a controversy over the ability of peer-review committees to identify applications for game-changing research, Agha says. "The criticism is that committees may be good at weeding out bad proposals, but are they really good at identifying great ones, and do they tend to reward more conservative projects?" she says.
Since the study focused strictly on projects that were awarded grants, it didn't directly address the question of whether the NIH panels are prone to turn down risky-but-high-payoff grant applications altogether.
However, the analysis demonstrated that among funded grants, the most highly rated projects do achieve significantly better results than their rivals in measures of groundbreaking science—including very high citation levels for their papers, publication in top scientific journals and the generation of patents.
Agha notes that this finding runs counter to the hypothesis that, as the rate of successful applications drops, peer reviewers fail to reward those risky projects that are most likely to be highly influential in their field.
She cautions that the analysis doesn't compare peer review to other research selection methods, and it doesn't imply that peer-review committees don't make mistakes or are completely unbiased. She also points out that scientists often complain about the time they invest in the peer-review panels.
But overall, Agha says, both the medical research community and the public at large "should be encouraged that peer reviewers are rewarding high-impact science."