A view on peer review
Scrutiny of scientific work is a fundamental pillar of Science. It is not only needed for correctness, in a purely technical sense, but also takes care of originality, novelty and significance, ensuring that the scientific work contributes to generating new knowledge. There are two stages where this task is especially important: before the start of a scientific investigation (projects) and when the investigation yields results (publications). At both stages the dominant (almost exclusive) method for evaluation is reviewing by peers.
A precursor of peer review began in England in the early 19th century, with referees commissioned by the Royal Society of London to write reports on manuscripts sent for publication in Philosophical Transactions [1]. Classical peer review is relatively young. It was questioned from the beginning and has never consolidated as a unique modelA standardized referee system developed from this point among the English scientific societies and slowly spread to independent journals and outside the Anglophone world. However, it was in the 1960s when refereeing propagated as the method to objectively judge Science (although the embracement of this system was delayed by some journals, like Nature, which adopted it in 1973) [1]. During that period of the 20th century the scientific activities experienced a huge expansion, especially in the USA, with a massive increase in public funding and a parallel social and political demand for objective methods to judge scientific quality. In fact, the term peer review was first used in evaluation procedures by American funding agencies of that time and was later used as a synonym of refereeing for the case of revision of publications. Thus, classical peer review, either of projects or papers, is a relatively young system. But perhaps most importantly, it has never consolidated as a unique solid model and was, from the beginning, put into question [1]. It is then not surprising that in current times, with technological changes affecting the ways of communicating Science, the future of peer review is subject of a particularly vivid debate.
Overflowed system
Today, the discussion about peer review connects with multiple other problems of Science [2] as well as with profound changes affecting modern publishing [3, 4]. With the dominance of bibliometrics as the measure to evaluate scientific performance, publishing has acquired a tremendous importance [5] and peer review, as the method employed to control what is published (and where), has become a crucial variable. Added to this, complex problems have emerged that affect directly to the quality of Science and thus uncover inefficiencies of the evaluation mechanisms, like poor reproducibility, fabrication of data, plagiarism or even direct corruption of the peer review system [6]. One would expect that the response of the scientific community to the clear need of stronger and more effective evaluation procedures would be to strengthen peer review, or even to evolve it into a new system, adapted to the challenges of the present and the future. However, in the vast majority of cases peer review continues being performed as it was conceived more than 50 years ago, despite its well known weaknesses.
Peer review, as the method to control what is published (and where), has become a crucial variable
The bibliometric epidemy [5] contributes to inflate the number of publications, and with that, the numbers of publishers and journals have also grown exponentially. This process is fuelled by digitalization and automation, which makes publishing easier, cheaper and quicker than ever. But can proper assessment, criticism and discussion of scientific work be similarly escalated and accelerated? Obviously not. First, this part of the business cannot be easily automatized and needs the cautious time and attention of expert human readers / evaluators. Furthermore, the scientific methodology is quickly changing and becoming more and more sophisticated. All branches of natural sciences (including, of course, Biophysics) are increasingly becoming multidisciplinary, meaning that diverse and well prepared experts are needed to judge the scientific work. With this requirements it is easy to visualize that the pool of possible (capable) referees must be of smaller size than the pool of their peer authors. In practice, the pool of reviewers is further reduced because of the lack of incentives to perform difficult, time constrained and barely recognized reviewing tasks. This all makes finding adequate referees an herculean exercise and accentuates the weakness of the system, in a vicious circle.
Blindness vs transparency?
In the scientific community there is consensus about the importance of a solid peer review system and the need to tailor its traditional scheme for current and future exigencies. However, there is no clear consensus yet about the actions to take –more than paying homage to the stoic anonymous reviewers (Fig. 1) [7].
A major discussion is settled about the convenient level of transparency [8]. First, we remind that the pioneer refereeing English system, as conceived and exercised originally by William Whewell [1], started being completely open, with reports published and signed by the referee (in a especial Proceedings journal). But it was soon realized that such an option prevented criticism and discouraged negative reports. Thus, after a couple of years of openness, refereeing became anonymous and reports were no longer published [1]. Today’s classical peer review is still mainly single blinded, with the identity of the authors know by reviewers, who remain unknown to the authors. This system has been attacked in two opposing directions. On the one hand, it is argued that the uncovered identity of authors facilitates possible discrimination because of gender, ethnic background, country of affiliation, personal relations or prestige of previous work. This has led to defend a double-blinded system [9], with unknown identities of both authors and referees. Although this option is preferred, according to some studies, it is recognized that is very difficult in practice to implement, since in many cases the type of work, cited references and other details can betray the identity of authors [10].
Openness is seen as positive: Fits with tendencies of modern publishing, brings transparency and provides ways to reward reviewers
On the other hand, it has been pointed out that a covered identity makes referees potentially immune from their possible unfair, harsh or unsound criticisms, which is used to defend a completely open system. Openness, at different levels, is gaining adepts as it fits well with new tendencies of open publishing and has a positive connotation, because is seen as a way to bring transparency to the publication process. It also can provide ways to reward the reviewers via publication of their reports along with authors’ manuscripts, which helps visualizing the usefulness of the peer reviewing process, even when the identity of the reviewer is kept anonymous. Although still a minority [11], an increasing number of journals are already experimenting with various degrees of openness. For instance, Nature Communications and the four journals published by EMBO offer the authors (and referees) the choice of open peer review, although the identity of the reviewers is not revealed. A number of medical journals like those from BioMed Central and BMJ have decided to go further and publish also the complete pre-publication and peer review history, including the name and affiliation of reviewers [8]. It is argued that this fully open model has the positive advantages of reviewers being more honest and constructive [12], although critics note that it may favour prestigious institutions from English speaking countries [13] or discourage criticism by junior researchers, who may fear retaliation by senior colleagues [14]. Despite this concerns, a majority of participants in the meeting on Transparency, Recognition, and Innovation in Peer Review in the Life Sciences, organized by ASAPbio, HHMI and Wellcome on February 7-9, 2018 at HHMI headquarters in Chevy Chase [14], (~81%, voting in person or through the internet) favoured the option of “publishing the content of peer reviews (with or without the reviewers’ names) and making these reports a formal part of the scholarly record with an associated DOI”. In line with this, a large survey published in PLoS ONE by the end of 2017 also shows support (~60%) among the scientific community to open peer review [15].
Expert scientists are busy people with exhausting responsibilities. Is “sense of duty” enough to involve them in peer review?
Incentives and rewards
We all know that the referee work is a voluntary “duty” with hardly any other motivation than the personal conviction to contribute to the soundness of one’s scientific discipline. But, almost by definition, the required expert scientists are very busy people under strong stress and with exhausting responsibilities at their institutions and research groups. Is “sense of duty” enough to involve them in the huge task of peer review?
Although a study promoted by Taylor & Francis Group concluded that “receiving free access to the Journal is the factor that would incentivise people most to review” [16], according to a recent survey [17], researchers ask for other incentives (Fig. 2). Particularly, they would like that peer reviewing activities are “taken into consideration when they are evaluated for grants, jobs or promotions.” The issue was among the ones discussed in the aforementioned ASAPbio meeting [14], where the majority of participants favoured a “formal recognition and credit for peer review activities from funding agencies and institutions, and acknowledging all contributors to a peer review report (such as students and postdocs) when submitting it to a journal.”
The need for proper peer review recognition is intensively debated [18, 19]. It has even been argued that free reviewing (and editorial work) is “not fair in ethical terms”, especially when publishing is such a profitable business [20]. Setting a standard mechanism for recognition of peer review activities is a main target of a partner initiative by F1000 and ORCID [21].
Meanwhile, Publons has created a data-base and a validation system to make possible that researchers who participate in peer review get credited from their effort [22]. The goal is double: On the one hand, it intends to work as a platform to share peer reviews and discussions, provided that their dissemination is not prohibited by the journals involved in the publications. On the other hand, researchers who register at Publons are able to show their record of verifiable peer review activities, so that they can get credit to be used in their individual evaluations, either for performance enquiries, promotions or grant applications. A few institutions, like Harvard, recognize already peer review and editorial activities, which must be reported in annual evaluations. This is a big step forward, but needs to extend worldwide in order to exert a significant impact.
In summary, there is little doubt that a deep improvement of the strategies for assessing the quality of Science is a most urgent need. However, as we just discussed, the general perception is that this crucial and demanding task is not sufficiently rewarded. Very interesting initiatives exist to adapt, at multiple levels, classical peer review. Although the changes already in place seem mostly experimental, we can foresee that a renewed refereeing system, characterized by increased openness, rewarding and recognition, will soon crystallize.
References
- Csiszar A. “Peer review: Troubled from the start.” Nature, 2016, 532: 306. DOI.
- Belluz J, Plumer B, Resnick B. “The 7 biggest problems facing science, according to 270 scientists.” Vox, sep. 2016. URL.
- Arns M. “Open access is tiring out peer reviewers.” Nature, 2014, 515: 467. DOI.
- Curry S. “Peer review, preprints and the speed of science.” The Guardian, sep. 2015. URL.
- Macilwain C. “Halt the avalanche of performance metrics.” Nature, 2013, 500: 255. DOI.
- Ferguson C, Marcus A, Oransky I. “Publishing: The peer-review scam.” Nature, 2014, 515: 480. DOI.
- Schiermeier Q. “Monument to peer review unveiled in Moscow.” Nature, may 2017. DOI.
- Pulverer B. “Transparency showcases strength of peer review.” Nature, 2010, 468: 29. DOI.
- Cressey D. “Journals weigh up double-blind peer review.” Nature, jul. 2014. DOI.
- Ware M. “Peer review: benefits, perceptions and alternatives.” Publishing Research Consortium, 2008, 4: 1.
- Parks S, Gunashekar S. “Tracking Global Trends in Open Peer Review.” RAND blog, oct. 2017. URL.
- DeCoursey T. “The pros and cons of open peer review.” Nature, 2006. DOI.
- Hopewell S, Collins GS, Boutron I, Yu L-M, Cook J, Shanyinde M, Wharton R, Shamseer L, Altman DG. “Impact of peer review on reports of randomised trials published in open peer review journals: retrospective before and after study.” BMJ, 2014, 349: g4145. DOI.
- Brainard J. “Researchers debate whether journals should publish signed peer reviews.” Science, feb. 2018. URL.
- Ross-Hellauer T, Deppe A, Schmidt B. “Survey on open peer review: Attitudes and experience amongst editors, authors and reviewers.” PLOS ONE, 2017, 12: e0189311. DOI.
- “Peer review: a global view.” Taylor and Francis, 2015. URL.
- “Peer-Review Survey.” ASAPbio, 2017. URL.
- EDITORIAL. “Review rewards.” Nature, 2014, 514: 274. DOI.
- Curry S. “Peer review is essential to good science – it’s time to credit expert reviewers.” The Guardian, jun. 2017. URL.
- da Silva JAT, Katavić V. “Free editors and peers: squeezing the lemon dry.” Ethics & Bioethics, 2016, 6: 203. DOI.
- Haak L. “F1000 and ORCID Partner to Launch Standard for Citing Peer Review Activities.” ORCID blog, may 2015. URL.
- van Noorden R. “The scientists who get credit for peer review.” Nature, oct. 2014. DOI.