Research paper publishing sting reveals lax standards of many open-access journals

Oct 04, 2013 by Bob Yirka report
Peer review reviewed. Few journals did substantial review that identified the paper's flaws. Credit: 'Who's Afraid of Peer Review?' by John Bohannon, Science 4 October 2013: Vol. 342 no. 6154 pp. 60-65 DOI: 10.1126/science.342.6154.60

(Phys.org) —John Bohannon, contributing news correspondent for Science, the highly respected peer-reviewed journal has conducted what he calls a "sting operation" that reveals problems with open-access publishing journals. The results of his operation have been published in a Science News piece.

Over the past several years, a debate has arisen in the academic and research community. Should papers written by researchers be published as open-access (free for readers) or behind closed pay-walls? Some suggest papers written by researchers that use public funds should always be available for anyone to read without having to pay. Others insist that the only way to ensure quality and integrity is to have well respected journals accept only those papers worthy of publication and then to vet them via a peer review process. In his sting operation, Bohannon offers evidence of major flaws in the open-access publishing system.

The overall purpose of his sting operation was to reveal whether open-access journals were truly peer-reviewing articles as they were claiming. To find out, he wrote a bogus in which he claimed to have found a lichen-based wonder drug that showed cancer fighting capabilities. But, he of course didn't do any research, instead he made everything up. As part of doing so, he wrote his paper in a way that was certain to be rejected by any real peer-review process—it was full of technical errors, and even went so far as to suggest bypassing clinical trials. Bohannon even had the paper reviewed by some experts in the field to ensure that the errors he introduced were so blatant that no respected peer group could miss them. Once the paper was ready, he used a computer program to create hundreds of slightly different versions of the paper (with different author names, etc.) and then sent them to 304 open-access journals asking for the paper to be considered for publishing. Sadly, nearly half of the targets accepted the bogus paper indicating that they had not peer-reviewed the paper or their reviewers were very low quality. In all he reports that 157 journals accepted the paper while 98 rejected it outright and only 36 responded back to him with comments suggesting they had actually read what he'd sent them. Furthermore he found that many of the journals that presented themselves as western based, were actually operating in India and other eastern countries.

Bohannon doesn't offer any real analysis of his sting operation, preferring apparently to let the results stand on their own with the clear implication that because of unscrupulous operators, open-source publishing is seriously flawed and in many cases should not be taken seriously by those wishing to publish research papers.

Explore further: University of California adopts open-access policy for research papers

More information: 'Who's Afraid of Peer Review?' by John Bohannon, Science 4 October 2013: Vol. 342 no. 6154 pp. 60-65 DOI: 10.1126/science.342.6154.60

See also: Flawed sting operation singles out open access journals

Related Stories

Recommended for you

F1000Research brings static research figures to life

13 hours ago

F1000Research today published new research from Bjorn Brembs, professor of neurogenetics at the Institute of Zoology, Universitaet Regensburg, in Germany, with a proof-of-concept figure allowing readers and reviewers to run ...

How science can beat the flawed metric that rules it

15 hours ago

In order to improve something, we need to be able to measure its quality. This is true in public policy, in commercial industries, and also in science. Like other fields, science has a growing need for quantitative ...

User comments : 19

Adjust slider to filter visible comments by rank

Display comments: newest first

Doug_Huffman
1.7 / 5 (12) Oct 04, 2013
Alan Sokal's warning ignored; in Transgressing the Boundaries: Towards a Transformative Hermeneutics of Quantum Gravity, Social Text Spring/Summer 1996 "Science Wars" issue.

Believe nothing read or heard without verification unless it Weltanschauung congruent. Does one credulously read comic books or take instruction from cartoons, is Science or Nature any different? A koan; is it clickbait? Follow the money.
Tom_Andersen
1.5 / 5 (11) Oct 04, 2013
Hilarious.

Lets let the paid journals research this.

I hope that entire publications of Science are indeed elaborate 20 year+ sting operations.
ar18
3.5 / 5 (6) Oct 04, 2013
Science, in a nutshell, is about finding flaws in your methodology, logic, techniques, etc, and this "study" by John Bohannon itself is flawed and shouldn't have been accepted for publication. Flaw #1: open access is not just about peer review, it is about eliminating the price gouging being conducted by closed access journals. Flaw #2: closed access is not an example of peer review, its an example of review by a hand picked committee. Flaw #3: there's no way to ensure that a small group of arbitrary people who do the reviewing will be or will remain unbiased by profit or politics - and both are very heavily involved in this process today. Politics is especially troublesome because it results in the filtering of dissenting or unpopular ideas.

The process of peer review needs to be done by all of our peers, not bean counters, politicians, or even a small highly paid committee. Scientists shouldn't have their research "taxed" by closed access journals.
StevanHarnad
2 / 5 (3) Oct 04, 2013
To show that the bogus-standards effect is specific to Open Access (OA) journals would of course require submitting also to subscription journals (perhaps equated for age and impact factor) to see what happens.

But it is likely that the outcome would still be a higher proportion of acceptances by the OA journals. The reason in simple: Fee-based OA publishing (fee-based "Gold OA") is premature, as are plans by universities and research funders to pay its costs:

Funds are short and 80% of journals (including virtually all the top, "must-have" journals) are still subscription-based, thereby tying up the potential funds to pay for fee-based Gold OA. The asking price for Gold OA is still arbitrary and high. And there is legitimate concern that paying to publish may inflate acceptance rates and lower quality standards (as the Science sting shows).

Harnad, S. (2010) No-Fault Peer Review Charges: The Price of Selectivity Need Not Be Access Denied or Delayed. D-Lib Magazine 16 (7/8).
Doug_Huffman
1 / 5 (9) Oct 04, 2013
"Closed access" is awkward. Better and already in wide use is PAYWALLED.
cantdrive85
1 / 5 (14) Oct 04, 2013
"I have no trouble publishing in Soviet astrophysical journals, but my work is unacceptable to the American astrophysical journals."
[Referring to the trouble he had with the peer reviewers of Anglo-American astrophysical journals because his ideas often conflicted with the generally accepted or "standard" theories.] Hannes Alfvén

"The peer review system is satisfactory during quiescent times, but not during a revolution in a discipline such as astrophysics, when the establishment seeks to preserve the status quo." Hannes Alfvén
Benjamin1988
3 / 5 (2) Oct 04, 2013
I hope he declared "competing financial interests" at the end of the article.
NikFromNYC
1.6 / 5 (13) Oct 04, 2013
Marcott's climate hockey stick of 2013 that appeared in Science might as well have been a sting operation too! Shame on Science for picking on the little guys as it degenerates into a propaganda rag.

http://www.bishop...all.html

On Facebook, Michael "Hockey Stick" Mann did a Snoopy dance over this fraudulent "confirmation" of his work.
JohnGee
1.4 / 5 (12) Oct 04, 2013
"Closed access" is awkward. Better and already in wide use is PAYWALLED.

Are you seriously scolding people for not using clear language? Half your posts are goofy limericks with entirely made up words. Even this post of yours is awkward. Maybe you have a language problem? Maybe you are a clown?
Jotaf
1 / 5 (1) Oct 04, 2013
Why not both - i.e both free, both peer-reviewed? After all, the publishers don't pay the scientists for their peer-review anyway: they're parasiting on both scientific community, both the tax payers, who are paying it all.


Exactly! They present a false choice, between "open access and non-peer-reviewed" or "closed access and peer-reviewed". Since reviewing and editing is done by voluntary scientists, you can run a respected journal with little money involved. For example:

http://jmlr.org/statement.html
muhammad lal
1 / 5 (3) Oct 05, 2013
Science , in short, your way , logic, technology, and about finding fault , and John Bohannon Roy itself, "research" should not be accepted and published fault . Defect # 1 : open access not just to the peer review , it is done by a closed- access journal to remove price gouging . Defect # 2 : closed access , for example , not peer-reviewed , hand picked committee of the review sample . Defect # 3 : This review does any small group of people will be fair or profit , or due to political left , there is no way to make sure - and everyone is very heavily involved in this process are today . Popular oppose it , or as a result of the idea for filtering is particularly troublesome in politics .

Peer review process, our colleagues , no bean counter , politicians , and even a little higher salary must be done by all of the committee . Their study, the scientists at the closed- access journal " tax " is not necessary.
a301khan
1 / 5 (3) Oct 05, 2013
Now we have real and proven data and result of the quality of SOME OA journals (neglecting the fact that it was not compared with similar subscription based journals and other weakness of this study). Even though this experiment is 'not perfect', but I am so happy to see that it throwing light on the quality of 'screening and peer review service' of OA journals. I strongly believe that the scholarly publishers should work like 'strict gate keeper' by arranging honest and sincere peer review service. This is the main difference between a 'scholarly publisher' and a 'generic publisher' (who publishes anything). Other works like typesetting, proofing, printing, web hosting, marketing, etc are important but not unique for a scholarly publisher. (My personal opinion is that we should not waste time by debating--Good OA, --bad OA--good CA--bad CA, etc. It is the time to work. We must jump to more effectively analyses and use these huge precious data)

Part 1.
a301khan
1 / 5 (3) Oct 05, 2013
I know and strongly believe that Beall, being an academician-librarian, also gives the highest importance to this particular criteria than any other thing. I congratulate Beall that his theory has been experimentally proven by the Sting Operation of Science.

I know this sting operation is going to generate a huge debate and one group will try to find out the positive points and other group will try to prove it as bogus experiment. A simple endless and useless fight and wastage of time. It will be more important to find out some way to use this huge data more effectively.

Part 2
a301khan
1 / 5 (2) Oct 05, 2013
Now I have some suggestions and questions
1. How are we going to use these huge data generated by this year long experiment?
2. I request DOAJ, OASPA to do some constructive works by using this huge data.
3. Can we develop some useful criteria of screening OA publishers from the learning of this experiment?
4. Is there any way of rewarding the publishers, who properly and effectively passed this experiment (rejecting the article by proper peer review. I noticed some journals rejected due to 'scope-mismatch'. It is definitely a criterion of rejection. But it does not answer, if scope was matched, what would happen).

Part 3
a301khan
1 / 5 (3) Oct 05, 2013
5. I saw the criticism of Beall that 'he is ..trigger-happy'. It is now time for Beall to prove that he not only knows to punish the bad OA, but he knows to reward also somebody, if it intends to improve from the previous situation. Is there any possibility that this data can be used for the 'appeal' section of Beall's famous blog. Sometimes judge can free somebody depending on the circumstantial proof, even if he/she does not formally appeal. (Think about posthumous award/ judgment.) I always believe that 'reward and inspiration of good' is more powerful than 'punishment for doing bad'. But I also believe that both should exist.

Part 4
antialias_physorg
not rated yet Oct 05, 2013
Why not both - i.e both free, both peer-reviewed?

For the simple reason: If you publish in a peer reviewed journal you sign away the rights to your paper to the journal (i.e. you don't own your own paper anymore).
You can't publish it anywhere else without being sued for copyright/ownership infringements.
indio007
1.7 / 5 (6) Oct 07, 2013
Why not both - i.e both free, both peer-reviewed?

For the simple reason: If you publish in a peer reviewed journal you sign away the rights to your paper to the journal (i.e. you don't own your own paper anymore).
You can't publish it anywhere else without being sued for copyright/ownership infringements.


Your actually being to nice in your description of the vast swath of property rights to are gifted to these journals. The journal gets free product to sell, the author gets zip.

I think the IRS needs to take a good look at this practice. Windfalls and gifts are supposed to be taxed at a high rate. I would bet they don't even claim the market value.
indio007
1 / 5 (5) Oct 07, 2013
In other news, soft drink makers release study demonstrating free rain water is a communist plot intended to destroy the market forces.
The study also found Mother Nature's hydrologic cycle to be "cutting corners" compared to modern processing.

Gimme a break. The journal Science must really be in trouble to use their supposed unbiased scientific reporting as a tool to eliminate market competitors.
Jotaf
1 / 5 (1) Oct 09, 2013
Actually, there's a subtler issue -- "author pays" open access, versus "no one pays" open access. "Author pays" may seem good at first, but presents the editors an incentive to have lax standards, and is still "for profit". The second is as free as it gets (and remember that reviewing is done by unpaid volunteers in all cases).

The link I posted above is to a journal where no money changes hands, and in my opinion they are preferred. However, the "author pays" model is much more common (no doubt because it's still "for profit"), and I guess that's what they sampled in the study. So the conclusion that they have more lax standards is not that surprising, but misrepresents the full spectrum of OA possibilities.