Decision making in scientific peer review unravelled with mathematical modelling

Professor Mikko Alava and undergraduate Tuomo Hartonen of Aalto University Department of Applied Physics have modelled the work processes and human decision making in scientific peer review with the help of statistical physics. Their study will improve understanding of how actions of reviewers and editors during the review work correlate with the decisions to publish or reject article manuscripts. The article presenting their research is now published in the journal Scientific Reports.

Peer review is one of the most pivotal guarantees of advancement in scientific knowledge and its objectivity. In practice, it comprises sending article manuscripts to journals, extensive commentary reading, corrections and suggestions, correspondence and decision making from scientists, reviewers and editors alike.

In their Nature's Scientific Reports article, Alava and Hartonen show that the time reviewers spend on working on a manuscript is clearly dependant on the perceived quality of the manuscript and on the decision whether to publish or to reject it in the end.

Alava and Hartonen study peer review as a series of dynamic processes, in which the reviewers' work phases and the decision making that comes with it must be modelled simultaneously. They have studied the life span of over 10 000 manuscripts in two physics journals all the way from submission to the final verdict of publication or rejection.

The work patterns and decision making are included in a mathematical model that takes heed of the varying tasks during the review process. Review work is always occurs in circumstances, in which many concomitant tasks and priorities mix. Alava and Hartonen call this the Deadline-effect: important tasks are completed amidst a constant friction caused by competitive tasks as a time limit approaches. Here decision making must be modelled as stochastic processes, that is, to mathematically manage the unpredictable randomness of task completion.

The amount of time that reviewers and editors take to process submitted articles depends on the perceived quality of the manuscripts. Prominent and poor manuscripts take a lot less time and work phases to go through peer review than the average majority. A manuscript that seems good at first glance features in the data, which Alava and Hartonen have analysed, as a stream of decisions markedly unaffected by the Deadline-effect and pressure.

Final verdicts made on manuscripts still define the statistics of peer review processes to such an extent that their integrity remains intact in the study. Alava and Hartonen conclude that expert reviewers make their decisions based on educated and informed guesses rather than on subjectively biased judgments.