December 23, 2014 report
Peer review could reject breakthrough manuscripts, study shows
(Phys.org)—A study by Kyle Siler of the Rotman School of Management at the University of Toronto and colleagues has found that well respected peer reviewed journals have rejected manuscripts that could discuss outstanding or breakthrough work. The researchers found that some manuscripts rejected by three leading medical journals went on to receive a large number of citations after publication in other journals. The study appears in the Proceedings of the National Academy of Sciences.
The peer review system serves as a gatekeeping system for scientific research, designed to ensure the publication of only the most well researched studies with the most important findings. Scientists depend on publication of their research in peer reviewed journals for career advancement. While peer review can prevent the publication of unimportant or poorly researched manuscripts, some scholars are concerned that it protects the status quo and suppresses innovation.
To evaluate this claim, Siler and his team studied a dataset of manuscripts submitted to Annals of Internal Medicine, British Medical Journal and The Lancet in 2003 and 2004. These journals rejected 946 of the 1,008 manuscripts in the dataset. 722 of the rejected journals never made it past the editor's desk and therefore, never even reached the peer review stage, at one or more of these three publications.
Other journals subsequently published 757 of the rejected manuscripts. The researchers looked at the number of citations these manuscripts went on to receive. They used the number of citations as a measure of quality, reasoning that when performing their own research, scientists usually choose to build on work they consider of good quality.
Siler's team found that, for the most part, editors and peer reviewers at the three elite journals did a good job of predicting the popularity of particular research papers among scientists. When the researchers assigned numerical scores to evaluations by peer reviewers, they found that, among both accepted and rejected papers, those with lower scores tended to receive fewer citations. Rejected manuscripts tended to receive fewer citations than accepted ones, and desk rejected manuscripts tended to receive fewer citations than those not rejected until the peer review stage.
However, the team discovered that some of the desk rejected manuscripts went on to receive many citations. The elite journals had rejected 14 of the most highly cited manuscripts and had desk rejected 12 of those.
The researchers acknowledge that the three journals may have rejected some of the manuscripts because they were more suited to specialist journals. Nevertheless, previous research suggests that peer review can incorporate bias, with reviewers basing decisions on the social characteristics of the authors or the intellectual content of the work. Gatekeepers tend to prefer work closer to their own and to favor the scientific status quo.
More information: Measuring the effectiveness of scientific gatekeeping, Kyle Siler, PNAS, DOI: 10.1073/pnas.1418218112
Peer review is the main institution responsible for the evaluation and gestation of scientific research. Although peer review is widely seen as vital to scientific evaluation, anecdotal evidence abounds of gatekeeping mistakes in leading journals, such as rejecting seminal contributions or accepting mediocre submissions. Systematic evidence regarding the effectiveness—or lack thereof—of scientific gatekeeping is scant, largely because access to rejected manuscripts from journals is rarely available. Using a dataset of 1,008 manuscripts submitted to three elite medical journals, we show differences in citation outcomes for articles that received different appraisals from editors and peer reviewers. Among rejected articles, desk-rejected manuscripts, deemed as unworthy of peer review by editors, received fewer citations than those sent for peer review. Among both rejected and accepted articles, manuscripts with lower scores from peer reviewers received relatively fewer citations when they were eventually published. However, hindsight reveals numerous questionable gatekeeping decisions. Of the 808 eventually published articles in our dataset, our three focal journals rejected many highly cited manuscripts, including the 14 most popular; roughly the top 2 percent. Of those 14 articles, 12 were desk-rejected. This finding raises concerns regarding whether peer review is ill-suited to recognize and gestate the most impactful ideas and research. Despite this finding, results show that in our case studies, on the whole, there was value added in peer review. Editors and peer reviewers generally—but not always—made good decisions regarding the identification and promotion of quality in scientific manuscripts.
© 2014 Phys.org