Strong scientific peer review leads to better science and policy formation

Jan 19, 2011

The current Special Issue of Technology & Innovation, Proceedings of the National Academy of Inventors is focused on the history, process and practice of scientific peer review, with several articles aimed at assessing scientific peer review within the federal government and peer review's relationship to federal policy formation.

According to A. Alan Moghissi, president of the nonproft Institute for Regulatory , and Michael S. Swetnam, of the Potomac Institute for Policy Studies, some confusion exists over what peer review is and how it should be used in policy decision making.

For Moghissi and Swetnam, advancements in science and technology drive an increase in the need for critical evaluation by the independent peers of those who create the advancements.

"Reviewers must be qualified and independent," they wrote. "Once more, review criteria must be identified, the process must be transparent, and over reliance on specific peer reviewers must be avoided. Too, institutional, intellectual and personal conflicts of interest must also be avoided."

Maintaining the integrity and improving peer review in the federal government was a key issue for Elmer Yglesias of the Science and Technology Policy Institute, Institute for Defense Analyses, who wrote that "there are indications that the (federal peer review) system may be over stretched and prone to error."

"Unprecedented high numbers of proposals and low funding rates may lead one to wonder whether the (federal) peer review system is up to the challenge," said Yglesias. "Indications are that it is over stretched."

According to Yglesias, NSF surveys indicate that reviewer's workload has increased and the number of reviewers per proposal has been declining, making funding decisions susceptible to error. The best solution to ensure review's integrity, he wrote, may lie in an Internet-based calibration system.

Nowhere may the process of peer review be more important than in determining what should be funded in military medical research. Dr. John F. Glenn, principal assistant for research and technology for the U.S. Army Medical Research and Materiel Command, made the point that candidate products may include prototypes for vaccines, drugs, diagnostics and other medical devices, information products, such as for training and behavioral interventions, as well as human performance information and information on a wide range of hazards.

"A distinctive feature of the core research programs is that they are end product-focused," wrote Glenn. "This leads to three separate roles for peer review – evaluation of research project quality, evaluation of research program quality, and independent validation of research products."

Important to Glenn is determining that a peer review process can validate that end products have met their objectives. He noted that the U.S. Food and Drug Administration's review process employs both internal and external scientific review panels and that the FDA's process is accepted as a substitute by the Department of Defense.

"Unfortunately, there is no comparable, consistently employed peer review process for the validation of medication information or clinical practice guidelines," wrote Glenn. "Such processes are needed as they are intrinsic for evidence-based medicine."

Not only does good peer review help validate a projects science, wrote lead author E. Melissa Kaine, MD, Captain, Medical Corps, U.S. Navy, it can also foster innovation.

According to Dr. Kaine, the Congressionally Directed Medical Research Program (CDMRP) is comprised of 18 individual programs seeking to eradicate diseases and support the 'warfighter.' The CDMRP emphasizes innovative, high-risk, high-gain research and employs a two-tier process to evaluate scientific merit, innovation and impact, followed by an external review. These processes, she said, support the mission to fund innovative research. While 'innovative' may be difficult to define, research that is transformative, paradigm shifting and that which represents more than an incremental advance in existing knowledge, serves as definitive.

One innovative aspect of the CDMRP review process is the use of layperson reviewers in addition to scientific experts.

"Consumer reviewers are included and involved at every stage of the funding cycle," she noted. "These reviewers are focused on the impact of the proposed project and provide valuable insight into a projects potential to result in positive outcomes."

Finally, the National Science Foundation (NSF) historian, Marc Rothenberg, reported on the history of peer review at the 60-year-old agency.

"NSF has had to consider 'broader impact' in addition to 'good science' in its mission to support basic research in the wide spectrum of science and engineering disciplines," wrote Rothenberg. "NSF has also fought a continuing battle to clarify criteria and to persuade peer communities to use the criteria as set down."

Rothenberg added that the trend at NSF since the 1960s has been to reduce the number of criteria, but to broaden the definition of those remaining.

Explore further: Best of Last Week – quantum pigeonholing, a hoverbike drone project and the sun goes quiet

More information: http://www.cognizantcommunication.com/filecabinet/Technology/techinnovation.html

Provided by University of South Florida

not rated yet
add to favorites email to friend print save as pdf

Related Stories

Federal peer review may be overstretched and error prone

Jan 10, 2011

The federal peer review system, by which research proposals are judged worthy for funding, may be "over stretched" and "susceptible to error," said Elmer Yglesias, a researcher at the Science and Technology Policy Institute ...

To publish or not to publish? That is the question

May 21, 2010

For more than 50 years medical research has been vetted through the peer-review process overseen by medical journal editors who assign reviewers to determine whether work merits publication. A study published in PLoS One invest ...

Peer Review Survey 2009: Preliminary findings

Sep 08, 2009

Should peer review detect fraud and misconduct? What does it do for science and what does the scientific community want it to do? Will it illuminate good ideas or shut them down? Should reviewers remain anonymous?

Science group posts interactive Web site

Dec 22, 2006

The San Francisco-based Public Library of Science says its online journal will post research and allow interactive review before and after publication.

Recommended for you

Local education politics 'far from dead'

18 hours ago

Teach for America, known for recruiting teachers, is also setting its sights on capturing school board seats across the nation. Surprisingly, however, political candidates from the program aren't just pushing ...

First grade reading suffers in segregated schools

18 hours ago

A groundbreaking study from the Frank Porter Graham Child Development Institute (FPG) has found that African-American students in first grade experience smaller gains in reading when they attend segregated schools—but the ...

Violent aftermath for the warriors at Alken Enge

19 hours ago

Denmark attracted international attention in 2012 when archaeological excavations revealed the bones of an entire army, whose warriors had been thrown into the bogs near the Alken Enge wetlands in East Jutland ...

Why aren't consumers buying remanufactured products?

21 hours ago

Firms looking to increase market share of remanufactured consumer products will have to overcome a big barrier to do so, according to a recent study from the Penn State Smeal College of Business. Findings from faculty members ...

Expecting to teach enhances learning, recall

21 hours ago

People learn better and recall more when given the impression that they will soon have to teach newly acquired material to someone else, suggests new research from Washington University in St. Louis.

User comments : 1

Adjust slider to filter visible comments by rank

Display comments: newest first

Moebius
not rated yet Jan 19, 2011
Too bad we don't have something similar for legislation. Or maybe we do? Laws made by idiots and reviewed by bigger idiots?