Strong scientific peer review leads to better science and policy formation

January 19, 2011

The current Special Issue of Technology & Innovation, Proceedings of the National Academy of Inventors is focused on the history, process and practice of scientific peer review, with several articles aimed at assessing scientific peer review within the federal government and peer review's relationship to federal policy formation.

According to A. Alan Moghissi, president of the nonproft Institute for Regulatory , and Michael S. Swetnam, of the Potomac Institute for Policy Studies, some confusion exists over what peer review is and how it should be used in policy decision making.

For Moghissi and Swetnam, advancements in science and technology drive an increase in the need for critical evaluation by the independent peers of those who create the advancements.

"Reviewers must be qualified and independent," they wrote. "Once more, review criteria must be identified, the process must be transparent, and over reliance on specific peer reviewers must be avoided. Too, institutional, intellectual and personal conflicts of interest must also be avoided."

Maintaining the integrity and improving peer review in the federal government was a key issue for Elmer Yglesias of the Science and Technology Policy Institute, Institute for Defense Analyses, who wrote that "there are indications that the (federal peer review) system may be over stretched and prone to error."

"Unprecedented high numbers of proposals and low funding rates may lead one to wonder whether the (federal) peer review system is up to the challenge," said Yglesias. "Indications are that it is over stretched."

According to Yglesias, NSF surveys indicate that reviewer's workload has increased and the number of reviewers per proposal has been declining, making funding decisions susceptible to error. The best solution to ensure review's integrity, he wrote, may lie in an Internet-based calibration system.

Nowhere may the process of peer review be more important than in determining what should be funded in military medical research. Dr. John F. Glenn, principal assistant for research and technology for the U.S. Army Medical Research and Materiel Command, made the point that candidate products may include prototypes for vaccines, drugs, diagnostics and other medical devices, information products, such as for training and behavioral interventions, as well as human performance information and information on a wide range of hazards.

"A distinctive feature of the core research programs is that they are end product-focused," wrote Glenn. "This leads to three separate roles for peer review – evaluation of research project quality, evaluation of research program quality, and independent validation of research products."

Important to Glenn is determining that a peer review process can validate that end products have met their objectives. He noted that the U.S. Food and Drug Administration's review process employs both internal and external scientific review panels and that the FDA's process is accepted as a substitute by the Department of Defense.

"Unfortunately, there is no comparable, consistently employed peer review process for the validation of medication information or clinical practice guidelines," wrote Glenn. "Such processes are needed as they are intrinsic for evidence-based medicine."

Not only does good peer review help validate a projects science, wrote lead author E. Melissa Kaine, MD, Captain, Medical Corps, U.S. Navy, it can also foster innovation.

According to Dr. Kaine, the Congressionally Directed Medical Research Program (CDMRP) is comprised of 18 individual programs seeking to eradicate diseases and support the 'warfighter.' The CDMRP emphasizes innovative, high-risk, high-gain research and employs a two-tier process to evaluate scientific merit, innovation and impact, followed by an external review. These processes, she said, support the mission to fund innovative research. While 'innovative' may be difficult to define, research that is transformative, paradigm shifting and that which represents more than an incremental advance in existing knowledge, serves as definitive.

One innovative aspect of the CDMRP review process is the use of layperson reviewers in addition to scientific experts.

"Consumer reviewers are included and involved at every stage of the funding cycle," she noted. "These reviewers are focused on the impact of the proposed project and provide valuable insight into a projects potential to result in positive outcomes."

Finally, the National Science Foundation (NSF) historian, Marc Rothenberg, reported on the history of peer review at the 60-year-old agency.

"NSF has had to consider 'broader impact' in addition to 'good science' in its mission to support basic research in the wide spectrum of science and engineering disciplines," wrote Rothenberg. "NSF has also fought a continuing battle to clarify criteria and to persuade peer communities to use the criteria as set down."

Rothenberg added that the trend at NSF since the 1960s has been to reduce the number of criteria, but to broaden the definition of those remaining.

Explore further: Science group posts interactive Web site

More information:

Related Stories

Science group posts interactive Web site

December 22, 2006

The San Francisco-based Public Library of Science says its online journal will post research and allow interactive review before and after publication.

Peer Review Survey 2009: Preliminary findings

September 8, 2009

Should peer review detect fraud and misconduct? What does it do for science and what does the scientific community want it to do? Will it illuminate good ideas or shut them down? Should reviewers remain anonymous?

To publish or not to publish? That is the question

May 21, 2010

For more than 50 years medical research has been vetted through the peer-review process overseen by medical journal editors who assign reviewers to determine whether work merits publication. A study published in PLoS One ...

Federal peer review may be overstretched and error prone

January 10, 2011

The federal peer review system, by which research proposals are judged worthy for funding, may be "over stretched" and "susceptible to error," said Elmer Yglesias, a researcher at the Science and Technology Policy Institute ...

Recommended for you

1 comment

Adjust slider to filter visible comments by rank

Display comments: newest first

not rated yet Jan 19, 2011
Too bad we don't have something similar for legislation. Or maybe we do? Laws made by idiots and reviewed by bigger idiots?

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.