Contents

  Freedom of Information Conference 2000

Fiona Godlee
BioMed Central

Peer review in the e-environment

It is a comfort to share a platform with people whose views you agree with so completely. However, there is a risk that if you repeat the same thing enough times you'll start to believe your own propaganda. We should think about that carefully.

A few weeks ago in the UK there was a public outcry when a politician suggested that the Queen might move out of Buckingham Palace. For those who don't know, Buckingham Palace is a huge piece of real estate in central London. Architecturally perfect for a museum but frankly impractical as a home for an elderly couple. The suggestion generated instant controversy and outrage in certain circles and the press had a field day. It was as if airing such views represented a threat to the fabric of our lives or even, God forbid, an end to polite society. The politician was eventually made to apologise, she retracted her suggestion, and we and the Queen can now rest safely in our beds. It is hard to imagine anything that would elicit the same reaction in the United States but perhaps I am not familiar with this country's sacred cows.

Changing peer review does not mean losing it

The sacred cow I am familiar with is scientific peer review. Peer review is, we would all agree, an essential thing with clear aims: to help decide which articles to publish, to look for ways to improve the quality of those articles, and to detect obvious errors of fact and interpretation. But peer review has become an institution, widely held in awe. From being a function that editors and scientists perform for practical purposes, it has become a thing in its own right, a guarantee, a stamp of truth, something that must not be scrutinised or questioned. So much so that in some people's minds we are forced to choose between classic peer review on the one hand and no quality control at all on the other. This has served to heighten people's resistance to the idea that peer review might change at all.

But it is clear that change it will. I want to highlight three developments that I think are promoting change and I will then go on to outline a vision of peer review in the future.

The first development changing peer review is the worldwide web. It brings with it opportunity and even, one might say, imperatives for greater openness and interactivity and I will talk about that in a minute when I talk about our plans at BioMed Central. The second thing we need to talk about is free access and that is why we are all here today. This will change peer review because it will force publishers to use it in a different way - in a way that adds real value to information. Some journals do this already but many more use peer review simply as a marketing tool. It allows them to make money from doing very little. Furthermore, what little these publishers do is subsidised by the academic community, through the voluntary labour of academic peer reviewers. An analogy is that you send your clothes to the laundry, they hold on to them for a year, blame the delay on volunteer launderers, ask you to buy the clothes back (often in a similar state to how they were sent in), but now the laundry has attached rules restricting where you can wear those clothes. Authors are beginning to see, and if they have not seen we are keen to tell them, that this is no longer a necessary ritual, there is an alternative.

The third development changing peer review is peer review research. This has put meat on the bones of growing concerns that peer review was not, after all, the objective and perfect process we all hoped it might be. Over the past fifteen years there has been a movement investigating biomedical peer review to see if it can stand up to the same rigorous scrutiny that it demands from science. I don't have time to summarise this research but, if you would forgive me, I want to indulge in a little bit of self-citation. There is now a book, courtesy of the BMJ, called Peer Review and Health Sciences and it summarises some of the evidence about peer review. What this evidence shows so far is that peer review, when done well, improves the quality and readability of scientific reports. But it also shows that peer review is very slow, takes up a lot of academic time, is highly selective, is very variable, arbitrary, prone to bias, open to abuse, patchy at detecting important methodological defects, and almost useless at detecting fraud and misconduct. One may say that peer review is so flawed that it is only the lack of an obvious alternative that keeps the process going.

Peer review's evolution

So how can peer review be improved and what will it look like in the new environment? I was going to talk about this in the abstract but several people have asked for more information about BioMed Central so I will use the short time available to describe the project and how I imagine its peer review process might work. BioMed Central is a commercial enterprise. It is a string of interlinked, peer reviewed, electronic journals spanning the whole of biomedical research. Its fundamental premise is that primary scientific research should be freely available to everyone. Attached to that is the premise that authors should retain control and copyright of their work.

Free access has several crucial implications to how BioMed Central will operate and the impact it will have on science. Firstly there will be a bias towards publication. Provided scientific results pass certain tests of validity, provided they are not incomprehensible and unethical, BioMed Central will publish them regardless of their originality or relevance. This decision is based on the belief that the results of science should be visible so that they are open to scrutiny and available for inclusion in systematic reviews and meta-analysis. It is also based on an understanding that much of what is published can be safely ignored.

The second implication of free access is that BioMed Central will provide an alternative to the current merry-go-round of submission, rejection, and re-submission to different journals, which wastes so much academic time. Evidence suggests that most papers will eventually be published somewhere but often after several rounds of peer review by several peer reviewers or, as Pat Brown told me yesterday, by the same reviewer (Pat), at the request of three different journals over the course of several years. This is not good use of anyone's time and has two unwelcome effects. It focuses the effort of peer review on the less good or less relevant work and it means that authors may become discouraged, get busy with other projects, and give up trying to get their work published. It's then placed in a bottom drawer. The impact of this on science is, as we know from the Cochrane Collaboration, that the cumulative record of science becomes distorted towards positive reports, and towards research about the more commercially rewarding treatments.

A third implication of free access is that BioMed Central will focus less on where and when reports are allowed to see the light of day and much more on the validation, selection, and summarising of published data for specific audiences. Paul Ginsparg called this the disentangling of distribution from validation. An example of this is a journal called ACP Journal Club which is an abstracting journal for clinical medicine. In its field, being selected for inclusion in ACP Journal Club provides a much more accurate and useful measure of the value of primary research than the blunt proxy measure of the impact factor of the journal in which the work was published.

We all know that journals have to fill their pages and this means that every issue of the New England Journal of Medicine, JAMA, and BMJ contain, as well as good stuff, not such good stuff. We have seen the data about how the impact factor of the journal does not always map to the quality of the individual articles. All the articles in a journal share the warm glow of that journal's impact value, regardless of their specific value. Secondary abstract journals like ACP Journal Club have clear criteria for selecting individual articles that are relevant and valid - in their case it is articles relevant to clinicians. The editor of the journal has found that less than five percent of all medical literature is relevant and valid to clinicians. That shows the size of the task these secondary journals take on board. These secondary sources are services that users may be willing to pay for and, in that light, perhaps present a business model for the future of peer review.

Briefly, some other key features of the BioMed Central model. It will reduce, we hope, the time taken to publish material by automating the process of matching articles to reviewers and perhaps even having same-day publication of accepted articles. In other words, although the human task of peer review will always be difficult to crunch down, the process at either side can be more efficient. BioMed Central will encourage and promote consensus building across journals on standardised formats for reporting research such as using CONSORT for randomised controlled trials and QUOROM for systematic reviews; these improve the quality of reports and will make it easier for authors as they move from platform to platform. BioMed Central will also provide a means for people to deposit and disseminate data of all sorts in the way that the Los Alamos physics archive does. So it will move away from the article as the only unit of publication. I should also mention that an important additional aspect of BioMed Central will be the register of controlled trials, providing for the prospective registration and unique numbering of clinical trials. The aim of these initiatives is to make it more difficult for data from trials to remain unpublished.

Open peer review

Finally, a brief word on open peer review. It is my intention that BioMed Central will have an open peer review process, but as this is my first week in the job I don't know exactly what form that will take. We need to discuss this with our advisers and users. My preferred model is one currently being validated in a trial I am involved in at the BMJ. Reviewers are randomised to either having their signed comments sent to the author or having their signed comments posted on the web alongside the accepted paper, the correspondence, and the initial submission so that the whole publishing process will, retrospectively, be visible on the web. Our hypothesis in the trial is that there will be no difference in the quality of the comments between the two groups of referees. What we do expect is that the reviewers who are asked to allow their information to be posted on the web may take a little more time and it may be that they will consider themselves being made more accountable to the scientific community.

There are increasingly compelling ethical arguments for open peer review and we now have good evidence that it is feasible. We know authors prefer it. There were fears in the beginning that junior researchers would be intimidated but it was notable that it was not the junior researchers saying this but the seniors. For me, an additional attraction of open peer review is that, as well as making peer reviewers more accountable, it becomes a way of incorporating peer review into the reward system of science. One can imagine a system where comments could be graded by the authors, the editors, and the users of the information. Open peer review merges with post publication peer review in the form of letters to the editor, changing peer review from a single binary event - to publish or not to publish- into a continuing mentoring process, and one in which the mentors will also get rewarded for the effort they put in.

Drummond Rennie has said that science works best in an environment of unrestrained criticism. My hope for BioMed Central is that it will create such an environment and that authors will embrace this alternative vision for science and send us their work.

Questions from the floor

Editors are as flawed

Question 1: I don't see peer reviewers as deciding which papers should be published. Peer reviewers advise the editor who makes the decision. As an editor I choose the peer reviewers and I have a stable of reviewers, not written down but in my head. I categorise them. Some do a thorough job, some are hyper critical, and some give the paper to a graduate student so the opinions are not really from the expert. I give a lot of leeway. If I liked the author I would send the article to an easy-going reviewer. So the attacks on the peer reviewers should really be aimed at the editors.

Fiona Godlee: Yes. I should have made clear that when I talk about peer review I mean the whole process. A journal has internal peer review and external peer review. When one evaluates the peer review process you want to try to account for those biases as much as possible. What you have described is that it can be a good process with good editors and reviewers, and it can be a bad process with less good editors and reviewers.

Will numerous versions complicate?

Questioner 2: People yesterday mentioned that manuscripts will go on the web as soon as they are accepted and will remain as manuscripts on those sites, even after the articles have been published. But if there are revisions, the authors are requested to make them and I am concerned about there being all these different versions available. How will readers know which is the real version, the final version? What is the point of leaving an accepted manuscript, peer reviewed but not edited, on the web even after the article has been published?

Fiona Godlee: The reason for putting the entire correspondence, the whole process, up there is simply so that one has a complete record. Very few people will want to access it but every so often there is a case when the process by which an article reaches the public is up for scrutiny.

Pat Brown: I don't see it as a problem in terms of ambiguity in respect to what is the most recent edition. I think it is obvious to mark the posted material. However, the other ambiguity is if a citation points to an earlier version and that version gets updated. How do you deal with that? I think that is straightforward. If each one has a unique identifier you have no problem knowing which version is being cited. Also you can alert readers to the fact that there is a newer version to the one being cited. I think having the full process available on line, even if it is rarely used, is a good thing.

Will openness defeat back scratching?

Question 3: I love the idea of open peer review but I don't think it will work because I think it will end up in mutual back scratching.

Pat Brown: I think the critical check on this is if the reviews are posted as a public document. I think people are not going to wimp out in terms of critiquing a lousy paper on a back scratching exercise because they will shame themselves.