The ins and outs of peer review

January 26, 2016 by Peter C. Doherty, The Peter Doherty Institute For Infection And Immunity, The Conversation
Every academic journal article is rigorously screened by other experts in the field. Credit: Shutterstock

If you are at all familiar with the operation of the Intergovernmental Panel on Climate Change (IPCC) you will know that, while the various authors are (unpaid) professionals of one sort or another with their own research programs, the IPCC itself neither commissions nor does research. Its conclusions are based solely on the information available in the published peer-reviewed scientific literature. What does that mean?

Every research paper that appears in print and/or online will have been read and critiqued (generally with complete anonymity) by at least two scientific peers. This mechanism was first put in place by the Royal Society in the 17th century, and has endured.

Sometimes, if the reviewers are at odds, a third or even fourth opinion will be sought. Though the tradition with law journals, for example, may be that the process is double blind – with neither the reviewers nor the authors being identified – the single-blind tradition in science means we generally know who wrote the paper, but can only speculate about the critics of our wonderful manuscript.

That may sound onerous but, in practice, it's not hard to get almost anything published at some level in what's broadly styled as the peer-reviewed , especially if it is well written and gives the appearance of having been done properly.

One checkpoint most editors will look for is that the paper comes from a university or research institute with some substance and that the author is not using, say, a gmail account. That doesn't mean that the article should be immediately rejected, but it provides some measure of upfront quality evaluation.

Academic employers have standards and, in a formal sense, most research grants are awarded to the institution, not to the individual researcher.

The list

What do I mean by "published at some level"? There's an A list of top journals – for instance, those with the word "nature" somewhere in the title – down to an E list or beyond to which researchers can drop if they are desperate to have an article appear in print and it has been rejected as it slid down the prestige ladder.

The leading journals, like Nature and Science, are read by a broad range of people from the physical to the biological sciences, while Nature Immunology and Nature Geoscience are obviously more specialised though of high stature.

In clinical medicine, the New England Journal of Medicine, The Lancet and the Journal of Clinical Investigation are very prestigious, while learned society publications like The Journal of Immunology and the Journal of Virology are thought of as "journals of record" for sound science within their particular specialty – or trade, if you like. Some of my most-cited papers are in these trade journals.

And there are journals for sceptics. Editors of that mindset will, for instance, select appropriately friendly reviewers. Otherwise, the intervention of a serious scientist sceptic who makes a substantial and well-supported case will generally be welcomed by one or other leading journal.

The reasons are obvious. Informed critique and the debate that results are illuminating and interesting to read. Those of us who are outsiders both learn from, and are entertained by, reading a lively discussion. There's nothing better than a good fight between intelligent protagonists! Then, nobody – and that includes – wants to persist with an erroneous view, so the alternatives must be aired.

Up for review

What of the anonymous, unpaid reviewers who work hard and do their best? Most are under a lot of pressure for all sorts of reasons, and they are likely to make the biggest effort with an article submitted to a high-profile journal.

With a demanding and detailed critique for a paper that does seem to have basic merit, the back and forth between authors, editors and reviewers asking for changes and/or new experiments can take a year or more. A problem with such a protracted process can be that, through a mix of exhaustion and dilution, what was a clear message will be largely obscured by the time the article appears.

Peer review should also be seen for what it is: an agreement between the editors, reviewers and authors that the article is now fit for general consumption by a quality audience, not a validation of infallibility, a concept that doesn't exist in science anyway!

What publication following actually says about a piece of work is that, so far as anyone who has been asked to critique it can detect, what looks to be an interesting study has been done using appropriate techniques, and the data looks to be both valid and to support the conclusions that are reached.

Beyond that, the reviewers should ensure that the methods used have been presented with sufficient clarity so that the study can be repeated, or at least understood, by others.

First step

Peer-reviewed publication is, in fact, just the first step in the validation of any important research finding. It's not an absolute; just, when done properly, the best evaluation that can be made without independent verification. What matters is that the findings and conclusions hold up as the field moves forward and that the work in question played a part in achieving a useful synthesis.

Even better, the new knowledge that appears first in a is truly validated when it has the power to provide, or to facilitate, a practicable solution to a real problem.

Even so, trying to stay both topical and ahead of the game does carry some inherent risks for editors. Many of the fraudulent papers that are eventually outed in, particularly, the biomedical sciences appear in what are generally thought of as top journals.

The reason is obvious. Editors are prepared to chance their arm to publish novel, high-impact findings. Sometimes the review process is at fault, but more often the fraud is carefully hidden.

Even so, though bad (or dishonest) science does not usually have any real long-term negative effect within science, it can cause immense damage if wrong ideas about subjects like childhood vaccination become established in the public consciousness. That's why it is so important that reviewers and editors do their best to see that potentially emotive, but erroneous, material does not seep through the process.

Explore further: Decision making in scientific peer review unravelled with mathematical modelling

Related Stories

Journal team adds reviewer pay to open-access model

January 18, 2015

A new open-access journal called Collabra plans to pay reviewers, and that's a twist in the world of scientific publishing. The reviewers get to exercise some options. They can keep the cash (generally a modest sum) or give ...

Nature journal to begin offering double-blind peer review

February 23, 2015

Well known and respected journal, Nature, will begin next month offering researchers who submit their work for peer review, the option of having it done via the double-blind method—whereby both submitters and reviewers ...

Open peer review could result in better quality of peer review

September 29, 2015

Whether or not a research article has been peer reviewed openly can seemingly make a difference to the quality of the peer review, according to research carried out by BioMed Central's Research Integrity Group and Frank Dudbridge ...

Recommended for you

An inflexible diet led to the disappearance of the cave bear

August 23, 2016

Senckenberg scientists have studied the feeding habits of the extinct cave bear. Based on the isotope composition in the collagen of the bears' bones, they were able to show that the large mammals subsisted on a purely vegan ...

Paleontologists discover major T. rex fossil (Update)

August 18, 2016

Paleontologists with the Burke Museum of Natural History and Culture and the University of Washington have discovered a Tyrannosaurus rex, including a very complete skull. The find, which paleontologists estimate to be about ...

Was 'Iceman Otzi' a Copper Age fashionista?

August 18, 2016

The 5,300-year-old Alpine mummy known as the Tyrolean Iceman died wearing leather clothes and accessories harvested from no less than five wild or domesticated species, a DNA analysis published Thursday revealed.

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.