Progress in peer review: Janne-Tuomas Seppänen discusses Peerage of Science

Posted by Biome on 3rd September 2013 - 5 Comments


Since November 2012, four BioMed Central journals – Frontiers in Zoology, BMC Biology, BMC Evolutionary Biology and BMC Ecology – have been ‘supporter’ journals of a new service for scientific peer review and publishing called Peerage of Science. This meant that manuscripts that had been reviewed through this initiative were encouraged, and that the Editors-in-Chief and Executive Editors of these journals would consider the accompanying reviews. The success of this innovative process has led to BioMed Central upgrading to full membership for these four journals. This now allows our Editors to extend publishing ‘offers’ to manuscripts submitted to Peerage of Science.

Janne-Tuomas Seppänen, Co-Founder, Managing Director of Peerage of Science, postdoctoral fellow, University of Jyväskylä, Finland.

To explain more about this initiative, and how our participation will benefit all involved with the peer review process, Biome interviews one of the co-founders of Peerage of Science, Janne-Tuomas Seppänen. Having founded Peerage of Science in 2011, alongside Janne Kotiaho and Mikko Mönkkönen, Seppänen is now managing director of the company. With the mission to fundamentally change how peer review is done, and to provide a service to authors, reviewers and publishers, Peerage of Science won the 2012 ALPSP (Association of Learned and Professional Society Publishers) Publishing Innovation Award. Seppänen is also an active scientist, and is currently an Academy of Finland postdoctoral fellow at the University of Jyväskylä, Finland.

 

Where did the inspiration for Peerage of Science come from?

There are many things that led to the idea, but here are some events I am consciously aware of.

Firstly, when I started getting my first reviewing requests as a fresh PhD some years ago, it felt like an initiation, sort of coming-of-age for a scientist. I put a lot of effort into trying to provide as thorough and justified evaluations as I could, and afterwards tried to enquire from editors whether my work was up to par, was it good or bad. Usually there was no response. The only yardstick was trying to compare my work to other reviewer’s reports on the same articles, but not all journals provided the opportunity to see other reviews, some did not even notify about the final decision.

Secondly, discussions (ok, rants) with other young postdocs about the high variance in the quality of peer reviews in the traditional system: sometimes they were thorough, justified, even brilliant, but then again too often hastily written, flippant, dismissive without analytical justification. People seemed resigned that even though that’s bad, there is no better system possible. I could not and cannot accept that.

Finally, the data in a paper by Pautasso and Schäfer in 2010 in Scientometrics about peer review delay made me realize that it’s not just the unlucky or inferior manuscripts of mine – peer review is very slow for everyone. When I introduced the Pautasso and Schäfer 2010 paper in our research groups gathering where we discuss current topics, there was a lively volley of responses and ideas being thrown around as usual. One sentence was ringing in my head when I went to bed that evening: “Someone should start a service”. I got up, writing and drawing flowcharts till sunrise, then went to describe the key concepts of Peerage of Science to Mikko Mönkkönen and Janne Kotiaho, who now are the other two founders of the company.

 

What makes it work?

The most important thing that makes Peerage of Science work is our peer-review-of-peer-review. The fact that reviewers know their report will be read, judged and scored by impartial colleagues, results in social pressure to justify arguments carefully, be thorough and fair, yet ruthlessly critical if the article does deserve a bashing. Astonishingly, even though the service provides almost absolute anonymity for reviewers, the kind of abuse of anonymity that sometimes occurs in the traditional system is nearly absent in Peerage of Science. The quality of course still varies, but that is now measured, and high quality work gets its due recognition and further motivates to build reviewer reputation.

There are a few other key concepts. To be able to submit manuscripts author must have positive ‘Review Balance’, which goes up by completing peer reviewing and down when your own manuscripts are reviewed by others. Unforgiving deadlines coldly enforced by code result in predictable and dependable schedules, as reviewers engage knowing they will be automatically dropped from a process without recourse if they miss a deadline. Free self-engagement of interested reviewers (once they’ve first passed our identity and validation process) naturally matches expertise with suitable papers. Structured and standardized limited-length ‘essays’ as peer review reports provide highly usable and comparable material, which both authors writing revisions, as well as editors making decisions, can utilize more efficiently.

Finally, the thing that makes Peerage of Science work is the feature available to editors: anonymous, silent tracking of any peer review process while being able to make a direct publishing offer to authors anytime, before they submit the work anywhere. A peer review process may be followed concurrently by many journals, and authors may receive more than one offer to choose from. From the publisher’s point-of-view, this is perhaps the most revolutionary aspect of the service.

 

How long is the peer review process in Peerage of Science?

The author specifies the deadlines for the four stages of peer review upon manuscript upload, and they usually pick our default values of ‘14 + 7 + 21 + 7’ days. But it is perhaps more interesting to look at how fast reviewers respond within the bounds of those deadlines.  On average, reviewers send in their essay in 15.2 days (±7.9 days std). Reviewers then get access to each other’s reviews and must score each other; a reviewer does that on average in 2.6 days (±1.8 days std). Authors then must revise the manuscript by the deadline they set for themselves in the beginning. Reviewers get notified about upload of the revisions and must then provide a final evaluation; on average a reviewer gets it done in 5.8 days (±4.9 days std).

What strikes me in these numbers is that the peer-review-of-peer-review gets done very fast, on average under half the time its deadline would dictate. It appears reviewers are curious to have a look at what other reviewers have said as soon as that becomes possible, and scoring other reviewers is a task they are eager to complete. Our rather unique and new cross-reviewing practice thus seems to be one of the best-working pieces of the system.

 

What sort of feedback have you received from the community?

Scientists are by nature open-minded but highly skeptical of any new idea, and that is reflected in some of the very early writings just after Peerage of Science launched in October 2011 (for example SciLogs and Trends in Ecology and Evolution). But mostly the early response was of curiosity and questions on how it works (for example Lab Times and INNGE).

During the almost two years since launch, I think both the scientific and the publishing communities have begun to see Peerage of Science as something that’s here to stay, rather than just curious novelty. We now get invited to give talks at publishing conferences and seminars (for example the forthcoming ALPSP Future of Peer Review seminar) as well as scientific workshops. Other new ventures like Altmetric have established collaborations with us, and even mainstream popular media occasionally mentions Peerage of Science when discussing academic publishing (for example the Economist).

One form of feedback we are very happy about are the two official accolades Peerage of Science has received:  the ALPSP Innovation Award and the Award by Communications Professionals of Finnish Universities.

 

Which peers are most active on Peerage of Science?

I think the demographics of the peer reviewers are probably very similar to the traditional system. About half of the reviews are done by professors and mid-career scientists; slightly fewer are done by postdocs.

But postdocs are clearly more active proportionally than others, as their proportion of the peer population is just 28.3 percent, yet they do 42.1 percent of the reviews.

Although reviewers have strict anonymity in any given peer review process, information on who has the right to review in the system is publicly available. Anyone can use a searchable map on our website to see who is in.

 

Do you see Peerage of Science broadening its scope to manuscripts outside of evolution and ecology?

Yes. In principle, manuscripts in any field of science can already be submitted to the system. The challenge is first having a critical mass of reviewers available. But authors can easily help with that themselves, first creating user accounts and inviting all of their own colleagues and asking those people to do the same for a few of theirs. If authors planning to submit articles manage to make invitations viral even for a short time, there can be a sufficient number of unaffiliated reviewers available from their field very quickly.

Expanding in evolution, ecology and conservation science was easier as the founders themselves are active scientists in those fields and recognized by name by many journal editors and colleagues. In other scientific fields Peerage of Science is more like just another start-up, and more effort is required in establishing trust and recognition.

Ideally, to get started in each new field it would help to first partner with one visionary highest-caliber journal of that field, with a mutually beneficial long-term agreement to get the ball rolling.

 

What are the main differences between Peerage of Science and other peer review services such as Axios Review and Rubriq?

If we look at the key concepts listed above under ‘What makes it work’, I believe they are all unique to Peerage of Science, and not available in other services. Most importantly, much of our philosophy rests on the concept of peer-review-of-peer-review, while other services are more editor-centric.

Peerage of Science features concurrent consideration by participating journals, any one of which can send a publishing offer to authors anytime. The other services seem to largely continue to operate within the confines of traditional sequential submission.

Also, Peerage of Science believes that publishers, in return for the revenue they make from publishing, want to continue to be the entity that makes peer review possible in terms of resources, and that scientific integrity is best guaranteed when reviewers are motivated by seeking academic recognition from their peers, rather than monetary payment from authors. Hence our business model is ‘publisher-pays’, while many of the other services operate on ‘author-pays’ models.

 

What’s next for Peerage of Science?

We want to promote respect for and appreciation of high-quality peer reviewing work. Towards this goal we are soon releasing an embeddable CV-widget, which peers can use to display their PEQ-scores (Peerage Essay Quality denoting the quality of their reviews) and number of reviews done in their own online CV’s and personal websites. Also, we will continue to award ‘The Peerage of Science Annual Reviewer Prize’ every year to promote peer reviewing as high-profile scientific work, rather than just a duty or a chore.

Looking to future, we are excited about Peerage of Science Commissions – a service currently being built on the foundation of our growing database. Any kind of organization needing paid scientific, cross-peer-reviewed peer review for larger projects, like for example academic book production, for evaluating grant applications, conference abstracts, doing research excellence assessments, or in hiring decisions, can have Peerage of Science arrange that evaluation. The idea is to relay paid professional opportunities for the best peer reviewers who earn their reputation in peer reviewing journal articles in the system, and offer an efficient one-stop solution for customer organizations.

To find out about the author experience of Peerage of Science, listen to our podcast with BMC Ecology author Jenny Dunn.