Federal peer review may be overstretched and error prone

January 10, 2011, University of South Florida

The federal peer review system, by which research proposals are judged worthy for funding, may be "over stretched" and "susceptible to error," said Elmer Yglesias, a researcher at the Science and Technology Policy Institute and author of "Improving Peer Review in the Federal Government," published in the current issue of TECHNOLOGY & INNOVATION, Proceedings of the National Academy of Inventors.

According to Yglesias, the federal peer review system is awash in an increasing number of funding proposals, leaving him to wonder if the peer review system is up to the challenge.

"Indications are that the system is over stretched," said Yglesias. "In addition, the number of U.S. researchers qualified to perform these reviews is not only limited, but declining as well. With fewer reviewers, funding decisions are more susceptible to error."

Not unlike a systematic check might be instituted for an engineering quality issue, Yglesias recommends a system of "calibration" to mitigate undesired outcomes. Calibration, he suggested, might prevent three kinds of errors: errors occurring because a proposal is selected for funding when it should not have been because of the reviewers failing to get good instructions and, second, errors occurring because reviewers were biased and swayed the panel. A third kind of error comes from over confident reviewers.

"This error occurs because some reviewers are correct far less than they think," commented Yglesias.

For Yglesias, the peer review system can be improved and validated through "calibration," which he defines as the use of specific measurement techniques compared to a standard. Providing standard examples to reviewers and running mock reviews would help, he added.

"Unfortunately, not many program officers are trained to facilitate a calibration," said Yglesias. "Also, it requires extra time and resources."

For a calibration program, he recommends "Calibrated Peer Review", a web-based program developed at UCLA through which student writing assignments are graded by student peers.

"It would not be difficult to conceive a similar system to calibrate the review of scientific proposals," he concluded.

More information: http://www.cognizantcommunication.com/filecabinet/Technology/techinnovation.html

Related Stories

Recommended for you

Chemicals in lavender and tea tree oil appear to be hormone disruptors

March 17, 2018
A new study lends further evidence to a suspected link between abnormal breast growth in young boys—called prepubertal gynecomastia—and regular exposure to lavender or tea tree oil, by finding that key chemicals in these ...

Exposure to low levels of BPA during pregnancy can lead to altered brain development

March 17, 2018
New research in mice provides an explanation for how exposure to the widely used chemical bisphenol A (BPA) during pregnancy, even at levels lower than the regulated "safe" human exposure level, can lead to altered brain ...

The coffee cannabis connection

March 15, 2018
It's well known that a morning cup of joe jolts you awake. But scientists have discovered coffee affects your metabolism in dozens of other ways, including your metabolism of steroids and the neurotransmitters typically linked ...

Smoking linked with higher risk of type 2 diabetes

March 15, 2018
The prevalence of diabetes has increased almost 10-fold in China since the early 1980s, with one in 10 adults in China now affected by diabetes. Although adiposity is the major modifiable risk factor for diabetes, other research ...

Key drivers of high US healthcare spending identified

March 13, 2018
The major drivers of high healthcare costs in the U.S. appear to be higher prices for nearly everything—from physician and hospital services to diagnostic tests to pharmaceuticals—and administrative complexity.

Pedometer health boost lasts four years

March 13, 2018
Wearing a pedometer to count your daily steps can keep you healthier and more active for as long as four years after using it, a new study shows.

1 comment

Adjust slider to filter visible comments by rank

Display comments: newest first

1 / 5 (1) Jan 10, 2011
The federal peer review system, by which research proposals are judged worthy for funding, may be "over stretched" and "susceptible to error,"...

Maybe. MAYBE???


Just perhaps it's because the never-ending, bottomless pit of taxpayer and borrowed dollars is a tempting target for "intellectuals", "professors", "economists" and lawyers. And with faceless, unelected bureaucrats determining who gets how much, is it any wonder that projects that "prove" the politically correct, like AGW, or white racism, or will reflect negatively on business and the free market, get tons of funding, and none go to the opposing views (plus of course the billions we spend building white elephants named after congresscritters.)

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.