The current Special Issue of Technology & Innovation, Proceedings of the National Academy of Inventors is focused on the history, process and practice of scientific peer review, with several articles aimed at assessing scientific peer review within the federal government and peer review's relationship to federal policy formation.
According to A. Alan Moghissi, president of the nonproft Institute for Regulatory Science, and Michael S. Swetnam, of the Potomac Institute for Policy Studies, some confusion exists over what peer review is and how it should be used in policy decision making.
For Moghissi and Swetnam, advancements in science and technology drive an increase in the need for critical evaluation by the independent peers of those who create the advancements.
"Reviewers must be qualified and independent," they wrote. "Once more, review criteria must be identified, the process must be transparent, and over reliance on specific peer reviewers must be avoided. Too, institutional, intellectual and personal conflicts of interest must also be avoided."
Maintaining the integrity and improving peer review in the federal government was a key issue for Elmer Yglesias of the Science and Technology Policy Institute, Institute for Defense Analyses, who wrote that "there are indications that the (federal peer review) system may be over stretched and prone to error."
"Unprecedented high numbers of proposals and low funding rates may lead one to wonder whether the (federal) peer review system is up to the challenge," said Yglesias. "Indications are that it is over stretched."
According to Yglesias, NSF surveys indicate that reviewer's workload has increased and the number of reviewers per proposal has been declining, making funding decisions susceptible to error. The best solution to ensure review's integrity, he wrote, may lie in an Internet-based calibration system.
Nowhere may the process of peer review be more important than in determining what should be funded in military medical research. Dr. John F. Glenn, principal assistant for research and technology for the U.S. Army Medical Research and Materiel Command, made the point that candidate products may include prototypes for vaccines, drugs, diagnostics and other medical devices, information products, such as for training and behavioral interventions, as well as human performance information and information on a wide range of hazards.
"A distinctive feature of the core research programs is that they are end product-focused," wrote Glenn. "This leads to three separate roles for peer review – evaluation of research project quality, evaluation of research program quality, and independent validation of research products."
Important to Glenn is determining that a peer review process can validate that end products have met their objectives. He noted that the U.S. Food and Drug Administration's review process employs both internal and external scientific review panels and that the FDA's process is accepted as a substitute by the Department of Defense.
"Unfortunately, there is no comparable, consistently employed peer review process for the validation of medication information or clinical practice guidelines," wrote Glenn. "Such processes are needed as they are intrinsic for evidence-based medicine."
Not only does good peer review help validate a projects science, wrote lead author E. Melissa Kaine, MD, Captain, Medical Corps, U.S. Navy, it can also foster innovation.
According to Dr. Kaine, the Congressionally Directed Medical Research Program (CDMRP) is comprised of 18 individual programs seeking to eradicate diseases and support the 'warfighter.' The CDMRP emphasizes innovative, high-risk, high-gain research and employs a two-tier process to evaluate scientific merit, innovation and impact, followed by an external review. These processes, she said, support the mission to fund innovative research. While 'innovative' may be difficult to define, research that is transformative, paradigm shifting and that which represents more than an incremental advance in existing knowledge, serves as definitive.
One innovative aspect of the CDMRP review process is the use of layperson reviewers in addition to scientific experts.
"Consumer reviewers are included and involved at every stage of the funding cycle," she noted. "These reviewers are focused on the impact of the proposed project and provide valuable insight into a projects potential to result in positive outcomes."
Finally, the National Science Foundation (NSF) historian, Marc Rothenberg, reported on the history of peer review at the 60-year-old agency.
"NSF has had to consider 'broader impact' in addition to 'good science' in its mission to support basic research in the wide spectrum of science and engineering disciplines," wrote Rothenberg. "NSF has also fought a continuing battle to clarify peer review criteria and to persuade peer communities to use the criteria as set down."
Rothenberg added that the trend at NSF since the 1960s has been to reduce the number of criteria, but to broaden the definition of those remaining.
Provided by University of South Florida
