Appraising the peer-review process

Jun 02, 2014 by Andre Spicer And Thomas Roulet, The Conversation
Credit: thierry ehrmann, CC BY

Most academic papers today are published only after some academic peers have had a chance to review the merits and limitations of the work. This seems like a good idea, but there is a growing movement that wants to retort as Albert Einstein did to such a review process.

Academic review process was different in Einstein's time. In his brilliant career, the only time his work was subjected to blind peer review – the authors don't know the reviewers and vice versa – he showed contempt for what is now the gold standard of science. Was Einstein right to be so suspicious of the peer-review process? Should we learn from him and begin to question the widespread use of peer review in academic science?

The first part of Einstein's career was in the German-speaking world. The German physics journals, in which Einstein published his breakthrough work, didn't have the same peer-review system we use today.

For instance, the Annalen der Physik, in which Einstein published his four famous papers in 1905, did not subject those papers to the same review process. The journal had a remarkably high acceptance rate (of about 90-95%). The identifiable editors were making the final decisions about what to publish. It is the storied editor Max Planck who described his editorial philosophy as:

To shun much more the reproach of having suppressed strange opinions than that of having been too gentle in evaluating them.

Many of the core scientific discoveries were not peer reviewed to modern standards. For example, the publication of the foundational paper describing the double helical structure of DNA by James Watson and Francis Crick in 1953 would have been jeopardised in the context of the classic review system as we know it, because of its speculative nature.

At the prestigious journal Nature, the peer-review system was only formally introduced in 1967. More recently, the discovery of distortion in gravitational waves by a telescope at Harvard – which has crucial consequences for our understanding of the formation of the universe – was presented as preliminary and treated with extreme caution and even sometimes with denigration, because it had not been peer-reviewed.

American adventure

It was only after Einstein came to the US in 1935 that he came face to face with the peer review process. He and his younger colleague, Nathan Rosen, sent a paper on gravitational waves to Physical Review, a journal which had established its reputation as the premier physics journal in the US. The paper had the potential to be highly controversial as it challenged the idea that gravitation was a wave.

John Tate, the editor of the journal, hesitated over Einstein's paper for a month. He then send it to a reviewer for comments – his selected reviewer was probably the famously gossipy Howard Percy Robertson, one of Einstein's colleagues at Princeton. The reviewer returned ten pages of comments which cast doubt on many of the central claims in the paper. The editor returned these comments to Einstein, asked him to consider the issues, and make any changes he saw necessary. Here is how Einstein reacted:

We (Mr. Rosen and I) had sent you our manuscript for publication and had not authorised you to show it to specialists before it is printed. I see no reason to address the – in any case erroneous – comments of your anonymous expert. On the basis of this incident I prefer to publish the paper elsewhere.

Although he withdrew the paper from Physical Review, Einstein went on to publish it in a much more low key outlet, the Journal of the Franklin Institute. However, the published version contains substantial revisions. It appears these revisions were largely on the basis of a discussion he had with Robertson at Princeton. The revised version toned down many of his original huge claims. These revisions may have saved him from public embarrassment.

What would Einstein say today?

Some might see this as an amusing historical incident. But we think it contains some important lessons for scientists of all kinds today. This is because it reflects the current tension regarding the peer-review system.

The story reminds us that double-blind peer review is only a relatively recent invention. For most of history of science, scientific advances were judged in a much more open and public fashion. It also shows us that the peer-review process can provoke displeasure among even the greatest. It can mean scientists not listening to criticism. Sometimes the result is that many ideas don't see the light of day.

These anecdotal lessons point to wider issues with the peer-review process, which itself hasn't been studied in much detail. The review process was meant to save scientists from mistakes and public embarrassment. The idea was that peers help to improve our work, and the review process of high-status journals can serve as "stamps of approval" or simply signal of quality.

But sometimes a collegial discussion rather than formalised peer review can be a better way of getting the message across. So far the peer-review process has been largely an item of faith – something that probably produces better science. However, there is a growing body of evidence which is challenging this notion.

An extensive review of the literature on peer review in 1998 identified problems. They found that there is a low level of reliability and agreement over the quality of submitted papers, largely because of a lack of objective evaluation criteria. Even worse, reviewers make mistake in their evaluation and often accept papers they should have rejected. As a direct consequence, established journals are usually biased against innovative work.

In our own field of management science, some have claimed the peer-review system means academic work can simply end up losing its integrity during the review process, and can result in trivial and boring research.

On a more positive note, when reviews are perceived of quality by authors, they tend to generate more citations, which is a measure of the number of times a research paper is mentioned in other journals and is considered a mark of quality. Also, reliability is not necessary for an efficient review process – often it is the process of peer review itself that contributes to improving the paper. Reviewers play a developmental role in the construction of knowledge, and the energy they deploy in this process is primarily driven by moral motives rather than any material interest.

Bad review for peer review

Perhaps the most gentle solution would be to improve peer review. There are clearly disagreements about how this might be done. Some claim the peer-review system needs to become more objective through the introduction of clearer criteria and better trained reviewers who are able to systematically apply these criteria. Others claim that some subjectivity is important because it can stop reviewers herding to established ideas, thereby crowding out alternative and often more innovative approaches.

The frustration regarding the peer-review system has lead to new hybrid systems to emerge. For instance, some scientific communities have experimented with making reviewing process public.

In the hard sciences, there are those who post papers online and other scientists decide whether they are worth being cited. PLOS ONE publishes any paper that has been considered as "technically sound" after a round of editorial review, and readers then judge the relevance of the research. Another alternative system would be to have a set of reviewers rating all the papers submitted online, and revising their judgement in case of resubmission. The growing number of open access journals has raised concerns that peer review would be progressively abandoned and search engines and metrics will replace editors and peer reviewers.

Let's try something new

Some, like Einstein before them, think that the peer-review system should be abandoned in favour of a "market of ideas" where the best research would naturally be identified by the crowd, hence reducing the cost of the review process. There are many potential dangers of these alternatives to peer review, the most obvious being expanded opportunities for "bad science" to masquerade as legitimate work. However, given the immense cost and frustrations associated with the process, we think it may be worth considering alternatives.

Peer reviewing is an important scientific institution. But there might need to be a range of forums in which scientific results and discussion takes place – peer-reviewed journals only being one among a number of options. Such options would then compete for both the attention of the readers and the best papers. We think this mixed scientific landscape would have pleased Einstein.

Explore further: Study examines development of peer review research in biomedicine

add to favorites email to friend print save as pdf

Related Stories

Peer review option proposed for biodiversity data

Oct 25, 2012

Data publishers should have the option of submitting their biodiversity datasets for peer review, according to a discussion paper commissioned by the Global Biodiversity Information Facility (GBIF).

Online game aims to improve scientific peer review accuracy

Nov 09, 2011

Peer review of scientific research is an essential component of research publication, the awarding of grants, and academic promotion. Reviewers are often anonymous. However, a new study by researchers at the Johns Hopkins ...

Recommended for you

How to win a Tour de France sprint

Jul 22, 2014

The final dash to the line in a Tour de France sprint finish may appear to the bystander to be a mess of bodies trying to cram into the width of a road, but there is a high degree of strategy involved. It ...

User comments : 24

Adjust slider to filter visible comments by rank

Display comments: newest first

Doug_Huffman
5 / 5 (1) Jun 02, 2014
LOL Crowd sourcing peer review is as silly as including a conversation as news on a science news aggregator. Best Valley-girl - Well, like YA, whatever!
swordsman
not rated yet Jun 03, 2014
If the reviewer doesn't fully understand the material being presented, then the paper will be rejected, and there will be no appeal. Anything would be better than this.
Writela
5 / 5 (1) Jun 03, 2014
The crowd sourced peer review could be hijacked and biased easily. It's not to difficult to make a good product from bad one or vice versa at the public forums. The contemporary peer-review process indeed introduces a multiple bias, but it's still the most reliable method of scrutiny we already know. Until the scientists are payed from public money, the peer-review should be institutionalized instead for to prevent the cheating and misconduct.
Writela
not rated yet Jun 03, 2014
The example of peer-review of Einstein & Rosen vs. Physical Review is controversial, just because we didn't find any gravitational waves so far. So it may be quite possible, that Einstein was actually correct with his initial stance and the peer-review was wrong. Or that the actual truth was somewhere inbetween.
TheGhostofOtto1923
1 / 5 (2) Jun 07, 2014
One more reason for AI. Humans are incapable of judging each other competently.
bluehigh
3.7 / 5 (3) Jun 07, 2014
Reminds me of this example as to how good AI is not ...

www.youtube.com/w...lbyTZsQY

Regardless of human fallibility and inadequate review systems, good science will always win through eventually.

Incosa
not rated yet Jun 07, 2014
..good science will always win through eventually.
Undoubtedly yes, because it's the truth, what always wins, not a science. As Max Planck has said, the science advances one funeral at a time. A new scientific truth does triumph because its opponents eventually die, and a new generation grows up that is familiar with it. And because the scientists are mortal, this process is undeniable...
After all, even the Holy Church accepted the Big Bang at the very end. It just did it, when it exhausted all other options, how to ignore it.
cantdrive85
1 / 5 (4) Jun 07, 2014
A couple of quotes from a pioneering Nobel Laureate who was considered an heretic and met much skepticism throughout his entire career, yet who is largely responsible for many ground breaking discoveries in plasma physics.

"The peer review system is satisfactory during quiescent times, but not during a revolution in a discipline such as astrophysics, when the establishment seeks to preserve the status quo." Hannes Alfvén

"I have no trouble publishing in Soviet astrophysical journals, but my work is unacceptable to the American astrophysical journals." Hannes Alfvén
Captain Stumpy
5 / 5 (2) Jun 08, 2014
Until the scientists are payed from public money,
@writela
your comment was good except for this part: there is no basis for this conjecture
you're making an assumption that all scientists are paid by what?
with that exception, I agree with your first post
Regardless of human fallibility and inadequate review systems, good science will always win through eventually
@bluehigh
I have to agree with this, even though there will be some who produce evidence of the occasional slip, overall it is the best process we have

The peer review process might need some work, but it is the best system we have. Perhaps it would be best if it went to blind peer review, so that a paper is not judged by the merit that others perceive the author has, but all the reviewers feedback should be visible to the author (except the reviewers names).
Incosa
1 / 5 (2) Jun 08, 2014
you're making an assumption that all scientists are paid by what
I didn't make such an assumption. Many scientists are payed from private moneys, but the areas of theoretical physics and cosmology aren't typical for it from obvious reasons: the potential return from such a research is so distant, that no private organization could utilize it. So I think, it's important to have some scientists payed from public money, but such a community poses another, opposite threat for unbiased research: simply the fact, they will research just for research, not for actual results of it. This is because all people are natural cheaters and when they're acting in crowd, their otherwise good private intentions are going into hell gradually. So that the careless funding of scientists for public money must be always balanced with privately funded research, which takes account to actual results, not just proclamation.
Incosa
1 / 5 (2) Jun 08, 2014
My perception of community of scientists is similar to any other gravitating system: when we put too much money into it, the scientists will start to aggregate around it naturally. The example of it was throwing of money into string theory research at the end of the last century or the CERN cooperation at LHC. The problem is, when we have too large pile of scientists dealing with common target, such a pile will start to live with its own life like selfish meme and it will lose a contact with neighboring reality like the black hole. Such a sectarian community will not only adjust itself to rules of reality, but it will adjust these rules to its own needs: for example it will start to define its own rules for peer review.
Incosa
1 / 5 (2) Jun 08, 2014
For example, CERN physicists refuse to apply peer review of their publications, claiming that the "external peer review is less stringent than our internal peer-review process" and that "only people "qualified" (i.e. checked for loyalty) to truly review the work are within the collaboration." They're publishing collectively, despite the list of authors exceeds many thousands of items - such cheating is indeed advantageous for most individuals, because scientists are honored for number of publications and their citations. CERN community is saying, it's "..a cognitive bubble that you can't escape - that you don't want to escape" - another typical sign of sectarian society, characterized by brain washing and sacrificing identity.
The community of string theorists evaded the review in similar way: it just publishes all articles at ArXiv.org directly without some review at all. Well - and we all paying this from our taxes...
Incosa
Jun 08, 2014
This comment has been removed by a moderator.
Captain Stumpy
5 / 5 (3) Jun 08, 2014
and that "only people "qualified" (i.e. checked for loyalty)
@zeph
assumption based upon ignorance and conspiracy theory
another typical sign of sectarian society, characterized by brain washing and sacrificing identity
more personal conjecture based upon ignorance and conspiracy theory. This is the problem with your post: there is NO FACT backing it up. perhaps you should consider doing an inside investigation to prove your point? There is obviously no other evidence out there for you to link to that is reputable
The dense aether model
is pseudoscience unless you find some empirical data supporting it and irrelevant to the topic as peer review is what called it crap based upon PHYSICS
Which is likely the REAL reason you are commenting here: because you are pissed that your pet religion has no basis in reality, nor has empirical data supporting it

cont'd
Captain Stumpy
5 / 5 (2) Jun 08, 2014
cont'd
I presume, this simple example illustrates what I mean about scale.
and I presume that you understand that given the lack of empirical data for DAW, there is NO REASON to bring it up unless you have some empirical data results that back it up?

You continue to post the same thing over and over again... this is not the first time you've claimed conspiracy and not been able to show any evidence (at least reputable evidence...)

When are you going to get it... just because you believe it, doesn't mean it is real.

unless there is empirical evidence supporting your conjectures, then you are posting unsubstantiated opinion which has all the same authority and validity as someone posting the proof that "Fairy excrement causes global warming"
Incosa
1 / 5 (3) Jun 08, 2014
more personal conjecture based upon ignorance and conspiracy theory. This is the problem with your post: there is NO FACT backing it up
What purpose the links in my posts are serving for, after then? You should prove first, they're irrelevant to my argumentation first. The whole thing is actually very simple: if you look around itself, you cannot see very few natural objects which follow strict laws of geometry: triangles, spheres. But if we increase the scope of view, then we suddenly realize, that the universe is full of regular spheres (large planets, stars, hydrogen orbitals). But if we increase this scope even more, then the regularity of Universe ceases again: the galaxies and large stars aren't already such regular objects anymore, the proton are elongated etc.

So if we want to apply schematic formal models, then we can succeed with it if we approach to the scale of large stars and atom orbitals, but once we move outside of it, our formal experience will fail again.
Incosa
Jun 08, 2014
This comment has been removed by a moderator.
antialias_physorg
5 / 5 (3) Jun 08, 2014
I'm not sure comparing Einsteins day to today makes much sense
1) There were far fewer papers published back then.
2) The number or researchers in any one specialty was much smaller. This would tend to bias peer review as it becomes much easier to identify who the paper is from (even if anonymized)
3) "The market of ideas" he envisioned was implicitly limited to people with scientific training, as others would not normally have access to the published material without making a major effort. In the age of the internet that would not be the case (and hence it wouldn't work to identify quality)

In our own field of management science

Management science? You have got to be kidding me. Notice how stuff like that wasn't even around in Einstein's day. If low level stuff like this gets to be called a 'science' then you bet your behind we need peer review. When the subject is already ephemeral then the papers published therein can't be anything but.

Incosa
Jun 08, 2014
This comment has been removed by a moderator.
antialias_physorg
5 / 5 (4) Jun 08, 2014
The formally thinking physicists

What does that even mean? More of your nonsensical blurb. You're like a politician/priest who uses a lot of words but doesn't say anything.

You have no clue how physicists think - because their IQ is so far above yours that you cannot even begin to guess at what goes on in a scientist's mind. You are like a toddler where they stride.
Captain Stumpy
5 / 5 (3) Jun 08, 2014
So, go ahead - the problem is open and cleanly defined for you. But if you cannot answer it, then please don't cross my circles, you're not even qualified for it.
@incosa-Zephir
first: I have posted proof debunking your aw/daw religious crap far too often, and you ignore it so...
I think Antialias says it best so I will just re-post his argument for an answer
More of your nonsensical blurb. You're like a politician/priest who uses a lot of words but doesn't say anything.

You have no clue how physicists think - because their IQ is so far above yours that you cannot even begin to guess at what goes on in a scientist's mind. You are like a toddler where they stride.
tell you what... after you learn REAL physics why don't you come back and re-read that aw/daw bull that you keep posting...

maybe then you will understand why it is considered PSEUDOSCIENCE

I noticed that some of your sock-puppets are being deleted off the site... too bad they dont IP ban you

Incosa
1 / 5 (4) Jun 08, 2014
have posted proof debunking your aw/daw religious crap
Which proof? Don't lie. I already noted many downvotes, but no arguments from you. Otherwise I perfectly understand, which physics today is considered pseudoscience and why. Before some time even the existence of meteorites and planes had been considered a pseudoscience with mainstream physics. It's just one-two hundreds old history. It's not so distant past - for example, the string theory is fifty years old already.
bluehigh
5 / 5 (1) Jun 08, 2014
Management science? You have got to be kidding me.


from a TV advertisement here ...

" Our washing machine scientists ... "

RealityCheck
1 / 5 (2) Jun 08, 2014
Hi CaptS. :) Very busy still, so can't stay/chat. Just saw your comments while reading-only through here today...
I have to agree with this, even though there will be some who produce evidence of the occasional slip, overall it is the best process we have...

...Perhaps it would be best if it went to blind peer review, so that a paper is not judged by the merit that others perceive the author has, but all the reviewers feedback should be visible to the author (except the reviewers names).

If "occasional slip" is disastrous, it is not just a "slip"; especially if (because of editor/peer bias/incomprehension etc) a correct person/idea is unfairly hounded/derided/ruined/dies/dismissed, and recognition delayed and orthodoxy remains 'uncorrected' for many years.

Anyhow, your above suggestion for improving 'science discourse' indicates you've come round to my own approach to 'doing science discourse'; ie: looking at objective merits of ideas, not at source/person etc. Good man! :)
Incosa
1 / 5 (2) Jun 08, 2014
Management science? You have got to be kidding me.
Never understimate softscience
Captain Stumpy
5 / 5 (4) Jun 09, 2014
Which proof? Don't lie
@zeph
I guess you forgot all about the conversations we had over this? another nail in the coffin of Aether religion
http://exphy.uni-...%202009.

or how about this one? remember it?
http://arxiv.org/...1284.pdf
Otherwise I perfectly understand, which physics today is considered pseudoscience and why
Blatant lie as you still support aw/daw like a religious acolyte looking for brownie points.
IOW- you've never been able to refute that experiment proves there is no aether, and I've pointed this out to you MANY times, which means: YOU ARE A BLATANT LIAR
indicates you've come round to my own approach
@rc
no... I never changed. I am an investigator, this has always been my approach.
IMHO a blind review has possibilities,but there is the lack of content (or personal) responsibility to contend with, too
Teper
Jun 09, 2014
This comment has been removed by a moderator.
Teper
Jun 09, 2014
This comment has been removed by a moderator.
Captain Stumpy
5 / 5 (4) Jun 09, 2014
I can see two links to two publication, but still no argument against anything
@teper-zephir
so what you are saying is that you don't understand the links?
Imagine that.
IOW - you don't know WTF you are talking about.

nice chatting with you. I didn't need you reaffirming your particular blind spot to empirical data, but it is nice to see that you are still the same old zephir... ignoring anything against his religion