Expert panels successfully rate medical research proposals, big-data analysis shows

April 23, 2015, Boston University
Credit: Charles Rondeau/public domain

The National Institutes of Health (NIH) is easily the world's largest funder of medical research, and outside scientists perform most of the research. Panels of these investigators also select the projects that the NIH supports. With the NIH budget slowly dropping, some experts have questioned whether this "peer review" process is prone to favoritism or to avoiding risky but potentially high-payoff studies.

Boston University (BU) and Harvard economists, however, have published an extensive analysis of NIH grants in the journal Science that shows a high correlation between how projects are rated by and the quality of the resulting .

"Peer review is tremendously important for determining what research is done in the United States, and the world, and yet we don't know very much about how effective these systems are," says Leila Agha, Assistant Professor of Markets, Public Policy and Law at the Boston University Questrom School of Business and co-author of the Science paper. "Our findings suggest that the process successfully identifies that are most likely to result in high numbers of publications and citations."

"Our work shows that peer review generates insights about the potential of research proposals that can't be predicted from past publications, grant histories, or other quantitative metrics," adds Danielle Li, Assistant Professor of Business Administration at Harvard Business School and the other co-author.

Agha and Li examined more than 130,000 research projects funded by the NIH from 1980 to 2008. They looked at how each grant application was rated and its follow-up accomplishments as indicated by the related scientific papers published, the number of times those papers were cited by other publications, and the patents that were awarded based on the work.

They discovered that applications with better peer-review scores are consistently associated with better research outcomes. That finding held true even when they took into account the field of research, the year the grant was awarded, the lead investigator's previous history of scientific papers and NIH grants, and other factors.

As NIH funding has become more competitive over recent years, and the percentage of funded proposals has fallen, there has been a controversy over the ability of peer-review committees to identify applications for game-changing research, Agha says. "The criticism is that committees may be good at weeding out bad proposals, but are they really good at identifying great ones, and do they tend to reward more conservative projects?" she says.

Since the study focused strictly on projects that were awarded grants, it didn't directly address the question of whether the NIH panels are prone to turn down risky-but-high-payoff grant applications altogether.

However, the analysis demonstrated that among funded grants, the most highly rated projects do achieve significantly better results than their rivals in measures of groundbreaking science—including very high citation levels for their papers, publication in top scientific journals and the generation of patents.

Agha notes that this finding runs counter to the hypothesis that, as the rate of successful applications drops, peer reviewers fail to reward those risky projects that are most likely to be highly influential in their field.

She cautions that the analysis doesn't compare peer review to other research selection methods, and it doesn't imply that peer-review committees don't make mistakes or are completely unbiased. She also points out that scientists often complain about the time they invest in the peer-review panels.

But overall, Agha says, both the community and the public at large "should be encouraged that peer reviewers are rewarding high-impact science."

Explore further: Analysis of peer review offers insights into research productivity

More information: Big names or big ideas: Do peer-review panels select the best science proposals? Science, … 1126/science.aaa0185

Related Stories

Analysis of peer review offers insights into research productivity

September 4, 2014
In a paper published today in the journal PLOS One, investigators with the American Institute of Biological Sciences report findings from an analysis of the research output from a series of biomedical research grants funded ...

Researchers claim NIH grant process is 'totally broken'

December 6, 2012
(Medical Xpress)—John Ioannidis, a researcher at Stanford University has, along with graduate student Joshua Nicholson, published a commentary piece in the journal Nature, taking the National Institutes of Health (NIH) ...

New study identifies gaps in NIH funding success rates for black researchers

August 18, 2011
Black scientists were significantly less likely than their white counterparts to receive research funding from the National Institutes of Health, according to an analysis of data from 2000 to 2006.

Recommended for you

How do babies laugh? Like chimps!

November 7, 2018
Few things can delight an adult more easily than the uninhibited, effervescent laughter of a baby. Yet baby laughter, a new study shows, differs from adult laughter in a key way: Babies laugh as they both exhale and inhale, ...

Tongue-in-cheek Nobels honor nutritional analysis of cannibalism, roller-coaster kidney stones treatment

September 14, 2018
A nutritional analysis of cannibalism and treating kidney stones on roller-coasters were research projects honored by tongue-in-cheek awards at Harvard University Thursday, designed to make you laugh first, and think later.

Pediatric robot patient offers new level of realism for doctors in training

September 10, 2018
A team of researchers and engineers at Gaumard Scientific has unveiled a new robot that raises the bar on medical training devices. The robot, called HAL, has been made to look like a five-year-old male patient and offers ...

Why men say they've had more lifetime sexual partners than women

July 25, 2018
The disparity between the number of sexual partners reported by men and women can largely be explained by a tendency among men to report extreme numbers of partners, and to estimate rather than count their lifetime total, ...

Censors jump into action as China's latest vaccine scandal ignites

July 22, 2018
Chinese censors on Sunday deleted articles and postings about the vaccine industry as an online outcry over the country's latest vaccine scandal intensified.

Revenge of a forgotten medical 'genius'

June 30, 2018
It's not an uncommon fate for a pioneering scientist: languishing unrecognised in his time before dying in obscurity. But as his 200th birthday approaches, the life-saving work of a Hungarian obstetrician is finally getting ...


Adjust slider to filter visible comments by rank

Display comments: newest first

not rated yet Apr 23, 2015
Thus maintaining the status quo. IS THAT GOOD?
not rated yet Apr 24, 2015
Thus maintaining the status quo. IS THAT GOOD?

What status quo? That good research should be preferred over bad?

Yep: that's good.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.