Study explores effects of review setting on scientific peer review

August 7, 2013

Research findings published today in PLOS ONE report that the setting in which a scientific peer review panel evaluates grant applications does not necessarily impact the outcome of the review process. However, the research found that the average amount of discussion panelists engage in during the review is reduced. The investigation examined more than 1,600 grant application reviews coordinated by the American Institute of Biological Sciences Scientific Peer Advisory and Review Services (AIBS SPARS) on behalf of a federal agency over a four-year period.

The researchers compared two years when grant applications were reviewed using an in-person peer review panel to two years when panels were conducted via or .

Funding organizations routinely bring experts together to review research grant applications. A process known as scientific peer review, the goal of these panels is to identify the best research applications.

"There are no studies exploring whether the review setting significantly alters the quality of the peer review process," stated Dr. Stephen Gallo, the lead author of this study and Technical Operations Manager for AIBS SPARS.

"Our goal is always a reliable and high-quality peer-review process. It is important that we understand the strengths and weaknesses of different peer review methods," said Scott Glisson, Director of AIBS SPARS and an author of this study.

The findings appear in "Teleconference Versus Face-To-Face Scientific Peer Review of Grant Applications: Effects on Review Outcomes" published in PLOS ONE.

"Little difference was found in most of the review metrics between face-to-face and teleconference settings," said Gallo. Application scoring was only modestly affected and reviewers used the full scoring range regardless of review setting. The reviewer ratings were highly reliable in both settings.

Often, the greatest anticipated difference between in-person and teleconference panels is the amount of time allocated to discussing applications. This study found teleconference or videoconference panels allocated less time to application discussions than in-person panels.

"This is a first of its kind study that provides valuable data to help research program managers select appropriate models for conducting peer review," said Glisson.

More research is needed. "We should know whether the reduced amount of discussion and peripheral interactions that occur in a teleconference setting influence the final panel outcomes, and, ultimately the productivity of the research that is funded," said Gallo.

Explore further: Federal peer review may be overstretched and error prone

Related Stories

Federal peer review may be overstretched and error prone

January 10, 2011

The federal peer review system, by which research proposals are judged worthy for funding, may be "over stretched" and "susceptible to error," said Elmer Yglesias, a researcher at the Science and Technology Policy Institute ...

Online game aims to improve scientific peer review accuracy

November 9, 2011

Peer review of scientific research is an essential component of research publication, the awarding of grants, and academic promotion. Reviewers are often anonymous. However, a new study by researchers at the Johns Hopkins ...

Genome assembly in the spotlight

July 23, 2013

The largest systematic assessment the process of genome assembly is published today in BGI and BioMed Central's open access journal GigaScience. The second Assemblaton competition saw 21 teams submit 43 entries based on data ...

Recommended for you

Ancient genome from Africa sequenced for the first time

October 8, 2015

The first ancient human genome from Africa to be sequenced has revealed that a wave of migration back into Africa from Western Eurasia around 3,000 years ago was up to twice as significant as previously thought, and affected ...

Rare braincase provides insight into dinosaur brain

October 8, 2015

Experts have described one of the most complete sauropod dinosaur braincases ever found in Europe. The find could help scientists uncover some of the mysteries of how dinosaur brains operated, including their intellectual ...

How much for that Nobel prize in the window?

October 3, 2015

No need to make peace in the Middle East, resolve one of science's great mysteries or pen a masterpiece: the easiest way to get yourself a Nobel prize may be to buy one.

The dark side of Nobel prizewinning research

October 4, 2015

Think of the Nobel prizes and you think of groundbreaking research bettering mankind, but the awards have also honoured some quite unhumanitarian inventions such as chemical weapons, DDT and lobotomies.


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.