Federal peer review may be overstretched and error prone

Jan 10, 2011

The federal peer review system, by which research proposals are judged worthy for funding, may be "over stretched" and "susceptible to error," said Elmer Yglesias, a researcher at the Science and Technology Policy Institute and author of "Improving Peer Review in the Federal Government," published in the current issue of TECHNOLOGY & INNOVATION, Proceedings of the National Academy of Inventors.

According to Yglesias, the federal peer review system is awash in an increasing number of funding proposals, leaving him to wonder if the peer review system is up to the challenge.

"Indications are that the system is over stretched," said Yglesias. "In addition, the number of U.S. researchers qualified to perform these reviews is not only limited, but declining as well. With fewer reviewers, funding decisions are more susceptible to error."

Not unlike a systematic check might be instituted for an engineering quality issue, Yglesias recommends a system of "calibration" to mitigate undesired outcomes. Calibration, he suggested, might prevent three kinds of errors: errors occurring because a proposal is selected for funding when it should not have been because of the reviewers failing to get good instructions and, second, errors occurring because reviewers were biased and swayed the panel. A third kind of error comes from over confident reviewers.

"This error occurs because some reviewers are correct far less than they think," commented Yglesias.

For Yglesias, the peer review system can be improved and validated through "calibration," which he defines as the use of specific measurement techniques compared to a standard. Providing standard examples to reviewers and running mock reviews would help, he added.

"Unfortunately, not many program officers are trained to facilitate a calibration," said Yglesias. "Also, it requires extra time and resources."

For a calibration program, he recommends "Calibrated Peer Review", a web-based program developed at UCLA through which student writing assignments are graded by student peers.

"It would not be difficult to conceive a similar system to calibrate the review of scientific proposals," he concluded.

Explore further: Damaging legacy: Mothers who smoke affect the fertility of their sons

More information: http://www.cognizantcommunication.com/filecabinet/Technology/techinnovation.html

Provided by University of South Florida

not rated yet
add to favorites email to friend print save as pdf

Related Stories

To publish or not to publish? That is the question

May 21, 2010

For more than 50 years medical research has been vetted through the peer-review process overseen by medical journal editors who assign reviewers to determine whether work merits publication. A study published in PLoS One invest ...

Peer Review Survey 2009: Preliminary findings

Sep 08, 2009

Should peer review detect fraud and misconduct? What does it do for science and what does the scientific community want it to do? Will it illuminate good ideas or shut them down? Should reviewers remain anonymous?

Breakthrough streamlines complex work assignments

Nov 16, 2005

Christodoulos Floudas and his students Stacy Janak and Martin Taylor have invented a mathematical formula that may transform the way that day-to-day work assignments are made across government and industry.

Recommended for you

User comments : 1

Adjust slider to filter visible comments by rank

Display comments: newest first

geokstr
1 / 5 (1) Jan 10, 2011
The federal peer review system, by which research proposals are judged worthy for funding, may be "over stretched" and "susceptible to error,"...

Maybe. MAYBE???

HaHaHaHahaHa

Just perhaps it's because the never-ending, bottomless pit of taxpayer and borrowed dollars is a tempting target for "intellectuals", "professors", "economists" and lawyers. And with faceless, unelected bureaucrats determining who gets how much, is it any wonder that projects that "prove" the politically correct, like AGW, or white racism, or will reflect negatively on business and the free market, get tons of funding, and none go to the opposing views (plus of course the billions we spend building white elephants named after congresscritters.)