Decision making in scientific peer review unravelled with mathematical modelling

April 22, 2013

Professor Mikko Alava and undergraduate Tuomo Hartonen of Aalto University Department of Applied Physics have modelled the work processes and human decision making in scientific peer review with the help of statistical physics. Their study will improve understanding of how actions of reviewers and editors during the review work correlate with the decisions to publish or reject article manuscripts. The article presenting their research is now published in the journal Scientific Reports.

Peer review is one of the most pivotal guarantees of advancement in and its objectivity. In practice, it comprises sending article manuscripts to journals, extensive commentary reading, corrections and suggestions, correspondence and decision making from scientists, reviewers and editors alike.

In their Nature's Scientific Reports article, Alava and Hartonen show that the time reviewers spend on working on a manuscript is clearly dependant on the perceived quality of the manuscript and on the decision whether to publish or to reject it in the end.

Alava and Hartonen study peer review as a series of dynamic processes, in which the reviewers' work phases and the decision making that comes with it must be modelled simultaneously. They have studied the of over 10 000 manuscripts in two physics journals all the way from submission to the final verdict of publication or rejection.

The work patterns and decision making are included in a that takes heed of the varying tasks during the review process. Review work is always occurs in circumstances, in which many concomitant tasks and priorities mix. Alava and Hartonen call this the Deadline-effect: important tasks are completed amidst a constant friction caused by competitive tasks as a time limit approaches. Here decision making must be modelled as stochastic processes, that is, to mathematically manage the unpredictable of task completion.

The amount of time that reviewers and editors take to process submitted articles depends on the perceived quality of the manuscripts. Prominent and poor manuscripts take a lot less time and work phases to go through peer review than the average majority. A manuscript that seems good at first glance features in the data, which Alava and Hartonen have analysed, as a stream of decisions markedly unaffected by the Deadline-effect and pressure.

Final verdicts made on manuscripts still define the statistics of processes to such an extent that their integrity remains intact in the study. Alava and Hartonen conclude that expert reviewers make their decisions based on educated and informed guesses rather than on subjectively biased judgments.

Explore further: To publish or not to publish? That is the question

More information: Read Tuomo Hartonen's and Mikko Alava's article 'How important tasks are performed: peer review' online.

Related Stories

To publish or not to publish? That is the question

May 21, 2010

For more than 50 years medical research has been vetted through the peer-review process overseen by medical journal editors who assign reviewers to determine whether work merits publication. A study published in PLoS One ...

Online game aims to improve scientific peer review accuracy

November 9, 2011

Peer review of scientific research is an essential component of research publication, the awarding of grants, and academic promotion. Reviewers are often anonymous. However, a new study by researchers at the Johns Hopkins ...

Federal peer review may be overstretched and error prone

January 10, 2011

The federal peer review system, by which research proposals are judged worthy for funding, may be "over stretched" and "susceptible to error," said Elmer Yglesias, a researcher at the Science and Technology Policy Institute ...

Recommended for you

The oldest plesiosaur was a strong swimmer

December 14, 2017

Plesiosaurs were especially effective swimmers. These long extinct "paddle saurians" propelled themselves through the oceans by employing "underwater flight"—similar to sea turtles and penguins. Paleontologist from the ...

1 comment

Adjust slider to filter visible comments by rank

Display comments: newest first

not rated yet Apr 22, 2013
Alava and Hartonen show that the time reviewers spend on working on a manuscript is clearly dependant on the perceived quality of the manuscript

Reason being: Peer review takes time. You don't get paid for it (i.e. as a reviewer you're actively spending money on it with every second you spend reviewing a paper).

Of course will you spend less time on a paper which, after first reading, seems marginally important as opposed to one that seems to contain genuinely innovative ideas.

Also when you get a paper for review you usually don't do it right away. Mostly you have other work to do as a researcher (more than you can fit into your work hours, assuredly). So review is one of those things that you put off 'until you have the time' (which usually means until a few days or the day before deadline).

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.