Expert panels successfully rate medical research proposals, big-data analysis shows

Credit: Charles Rondeau/public domain

The National Institutes of Health (NIH) is easily the world's largest funder of medical research, and outside scientists perform most of the research. Panels of these investigators also select the projects that the NIH supports. With the NIH budget slowly dropping, some experts have questioned whether this "peer review" process is prone to favoritism or to avoiding risky but potentially high-payoff studies.

Boston University (BU) and Harvard economists, however, have published an extensive analysis of NIH grants in the journal Science that shows a high correlation between how projects are rated by and the quality of the resulting .

"Peer review is tremendously important for determining what research is done in the United States, and the world, and yet we don't know very much about how effective these systems are," says Leila Agha, Assistant Professor of Markets, Public Policy and Law at the Boston University Questrom School of Business and co-author of the Science paper. "Our findings suggest that the process successfully identifies that are most likely to result in high numbers of publications and citations."

"Our work shows that peer review generates insights about the potential of research proposals that can't be predicted from past publications, grant histories, or other quantitative metrics," adds Danielle Li, Assistant Professor of Business Administration at Harvard Business School and the other co-author.

Agha and Li examined more than 130,000 research projects funded by the NIH from 1980 to 2008. They looked at how each grant application was rated and its follow-up accomplishments as indicated by the related scientific papers published, the number of times those papers were cited by other publications, and the patents that were awarded based on the work.

They discovered that applications with better peer-review scores are consistently associated with better research outcomes. That finding held true even when they took into account the field of research, the year the grant was awarded, the lead investigator's previous history of scientific papers and NIH grants, and other factors.

As NIH funding has become more competitive over recent years, and the percentage of funded proposals has fallen, there has been a controversy over the ability of peer-review committees to identify applications for game-changing research, Agha says. "The criticism is that committees may be good at weeding out bad proposals, but are they really good at identifying great ones, and do they tend to reward more conservative projects?" she says.

Since the study focused strictly on projects that were awarded grants, it didn't directly address the question of whether the NIH panels are prone to turn down risky-but-high-payoff grant applications altogether.

However, the analysis demonstrated that among funded grants, the most highly rated projects do achieve significantly better results than their rivals in measures of groundbreaking science—including very high citation levels for their papers, publication in top scientific journals and the generation of patents.

Agha notes that this finding runs counter to the hypothesis that, as the rate of successful applications drops, peer reviewers fail to reward those risky projects that are most likely to be highly influential in their field.

She cautions that the analysis doesn't compare peer review to other research selection methods, and it doesn't imply that peer-review committees don't make mistakes or are completely unbiased. She also points out that scientists often complain about the time they invest in the peer-review panels.

But overall, Agha says, both the community and the public at large "should be encouraged that peer reviewers are rewarding high-impact science."

Explore further

Study investigates conflict of interest in biomedical research proposals

More information: Big names or big ideas: Do peer-review panels select the best science proposals? Science, … 1126/science.aaa0185
Journal information: Science

Provided by Boston University
Citation: Expert panels successfully rate medical research proposals, big-data analysis shows (2015, April 23) retrieved 2 July 2020 from
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.

Feedback to editors

User comments