Forum index

Various ideas about scientific publishing

Etienne Joly
Toulouse, France
email : atn@cict.fr

Abstract

For the benefit of the scientific community, completely Open Access to all primary scientific articles is clearly the only way to go. But to ensure the quality of the papers published, it is hard to conceive that scientific publishing could be carried out by others than money-earning professionals. The only viable solution is therefore for the publishing charges to be levied on the authors. This is in fact very much the route followed by the pioneering enterprise launched as Bio Med Central. On the whole, however, authors have very understandably been reluctant to publish their first rate papers in such journals because of negative perceptions and upfront charges for publication.

I believe, however, that it would be possible to set up a system whereby papers would get evaluated for publication solely on their scientific soundness, whilst the best papers would still get recognised and their authors rewarded for making important contributions. For example I would envisage that the amount charged for the publication of their manuscript would be inversely related to the scientific impact of that paper. The ground basis of this proposal is that papers would be rated retroactively, and this rating would provide the authors with a quotable evaluation of their publications that could be used on their CVs or their grant applications.

In this slightly idealistic scheme, the scientists' main concern would be to produce the best science they can, and to deliver it in the best and most complete format to fellow scientists, and not to seduce editors and/or referees. Since acceptability would be based on soundness, not on the evaluation of the interest of the results described, the referee's role would be to help produce the best possible articles. And because scientists would be the payers, scientific publishers would have to compete with one another by providing the best possible service at the best prices to the scientific community.

Background: Commonly shared views and opinions on publishing scientific papers.

A) With the development of electronic publishing, the current situation where scientists pay to publish the results of their research, and pay again to access the journals where it is published, has become inappropriate, and calls for radical changes

B) The event of electronic publishing, by removing the printing and shipping costs which previously justified the charging of hefty subscriptions, means that publication of scientific data should rapidly evolve towards a completely Open Access mode. Scientists are both the producers and the users of scientific literature, and in both roles the evolution towards a completely Open Access publication policy is clearly advantageous: writers want to be read by as many people as possible, and readers want to have access to as much published material as possible.

C) Researchers spend too much time trying to publish their research in the best journals possible, and too little time doing research. On the same line of thought, there is too much emphasis on trying to "sell" papers to the best journals with the minimal amount of data, rather than making sure that the papers tell as solid and complete a story as possible.

D) At the same time, it is well recognised that truly innovative scientific contributions have often been rejected from the first journal where they were submitted for publication. Groundbreaking ideas are quite often ill perceived by the scientific establishment. The only way to judge the importance of scientific contributions is retrospectively.

What needs to evolve and what needs to be preserved

Over the last 10 years, the development of electronic publishing has considerably speeded up the publication process and would allow the automatic evaluation of the impact of individual publications, rather than having to rely on the 'impact factor' of the journal that the authors managed to get published in. Peer review should not be about how interesting results are, but should simply ensure that papers that are published are scientifically sound and that they make appropriate reference to previously published works.

Perhaps the one crucial aspect of scientific publishing is to ensure that papers are well written and intelligible to the broadest possible range of readers. Most scientists are trained to carry out lab work, and are much better at designing and conducting experiments to further their research than at writing up their results for publication.

Diffusion of knowledge through scientific publishing is, and always will be a very important part of the process of scientific progress. For this reason, it should be handled by professionals, i.e. people whose job it is to diffuse scientific information, and who should not be ashamed of making money for doing their job well. To ensure the quality of the services provided, I believe that scientific publishing should remain a commercial operation, whereby publishers draw financial returns from their activity and compete with one another by delivering as good a service as possible, both to their authors and readers.

Groundbreaking, truly innovative research is based on creativity and on the capacity of scientists to either build upon or to lay flat pre-established dogmas. Research needs to be primarily curiosity driven, and it is my sincere conviction that purpose-driven research is seldom the way that real advances are made, and when they are, it is at very high cost to value ratios.

Scientific publishing stands in contrast with this since it relies very little on creativity, but much more upon carrying out its contributions by applying well-tested rules. The experience of the past 50 years has clearly demonstrated that high quality scientific publishing is far from being incompatible with commercial enterprise, and I believe that it should probably remain that way. The conditions have changed, however, and the publishing world needs to adapt to this. Scientists should keep doing research, and keep relying on money-making publishers to provide the professional service of broadly diffusing their findings in the best format possible.

Could a functional system be organised around all these constraints ?
A step-by-step description of how things could work:

1) When a paper is submitted, it would first be read by the journal's editorial staff, to ensure that it is reasonably well-written, grammatically sound and complies with the journal's recommended format. If this were not the case, the editor would contact the authors to address the problem, and offer either to improve it through in-house editing services (with costs to be charged to the authors) or direct them towards the services of third party copy-editors. This would take place before sending the manuscript to referees. Note that the fact that editors would actually read the entire paper as an initial step would already be a big improvement on the current situation, where most papers are sent to referees after the editor or a sub-editor has read only the title and sometimes the abstract.

2) The paper, once it has met these basic editorial requirements, would be sent to at least three referees, experts in the field who would be asked to address the following four questions:
a) Are the experiments scientifically sound ? (i.e. have the right controls been performed, are the methods used appropriate to answer the questions asked, are the results significant ?)
b) Are the conclusions drawn supported by the data ?
c) Are the data novel and have appropriate references been made to previously published work ?
d) What suggestions do you have for improving the readability and impact of the manuscript ?

My feeling is that the perceived role of referees needs to shift from the too common attitude of simply seeking to denigrate and slow down the work of their peers, towards a more positive attitude. Their main role should be one of constructive criticism, trying to make suggestions so that the results presented can be published in the most scientifically correct and meaningful format possible, thereby optimising the value of the work performed for the benefit of the whole scientific community. In an ideal world, I think that refereeing would not even need to remain anonymous.

For these reasons, I think it may be well worth considering the possibility of paying referees, according to the quality of the job they have done. I think it would be particularly interesting to allocate something like US$150 as fees for three referees, but in variable portions depending on the quality of the comments provided (and how promptly it was delivered etc.). The editors (possibly assisted by feedback from the authors themselves) would then distribute these monies - for example 50-50-50, or 75-75-0, or 100-50-0 or 150-0-0. In addition to being a tangible source of income for students and post-docs who so often end up dumped with such boring jobs of refereeing third rate papers, a researcher's capacity to referee papers constructively could even become something that they put on their CV (with bank receipts to prove it).

3) Once the comments from the referees are back with the editor, he/she would have to decide whether the paper is scientifically correct, or whether it needs to be modified and to what extent. The expected rate of outright rejection would presumably be extremely low (making it worth their while for the editors to read the papers before sending them to referees). The authors would then be asked to comply with these requests, or to provide arguments for not wanting to do so.

4) After professional copy-editing of the manuscript, the paper would then be published online. For five months, the paper would be open to a discussion forum for the whole scientific community via a web interface. Comments would first be read by the editor, before passing them onto the authors, with the option for the writers of the comments to remain anonymous.

The editor would have the option of publishing online any comments that contribute constructively to the paper. This would be another way that significant contributions, via constructive comments to work published by others, could be added to researchers' CVs. From the perspective of teaching students, the possibility of voicing constructive criticism to published papers would also represent a fabulous opportunity and incentive to liven up dreary journal clubs that take place in many research institutes. If, upon careful reading and examination of a paper, groups of students found that controls were missing, or that previously published work had not been appropriately taken into account, they could write up their comments and submit them online. The possibility of signing as 'a departmental journal club' would preserve their anonymity, which they may feel could be important for future job applications.

5) After five months, the forum would be closed, and the authors would be given one month to address the various comments received, either by modifying the original manuscript or by publishing a reply rebutting the objections to their paper.

6) Six months after initial publication of the paper, the final version of the paper would then be complete, potentially enriched by a discussion forum that would be added to the same webpage as the final paper.

7) At this time the paper's impact would be assessed, by counting how many times it has been cited in primary papers and in review articles over the six months since its initial publication. Counting how many times a paper has been viewed and downloaded could also be included in the evaluation. Another potential marker of esteem would be if the paper was rated in Faculty of 1000 (or an equivalent evaluating service), or had been the subject of a News and Views column in the magazine section of other scientific journals. The number of citations for individual papers can already be evaluated via the ISI services, but this is clearly far from being a perfect indication of the importance of an article. For example, certain research fields such as ecology are much less prone to being cited than others. Ultimately, I would envisage that citation datasets, such as the one collected by ISI, could permit an overall computerised assessment of scientific impact in comparison to papers in the same field, over a period of time. I am not aware of such studies having been carried out, but I am sure that the rate at which papers get cited would reveal different types of papers, such as "trendy shooting stars" for papers that get cited a lot over a short time, or "steady but sturdy" for papers that keep being cited a few times a year for several years. Visionary papers would probably be ignored for a while, before being recognised as such, after which their citations would rise suddenly.

The computer exercise of evaluating articles would therefore have to take place continuously, or at several times, say every 6 months for two or three years after publication. With such a system, authors of "visionary" papers would keep the right to appeal to the editors to get the evaluation of their paper reconsidered. If a paper that had been largely ignored for months or even years suddenly started being cited very frequently, that paper could easily be re-evaluated. It's grade would then be modified in accordance, and the author would not only receive the corresponding voucher (see below), but could modify the grade on job and grant applications.

The precise equations and algorithms resulting in the final score of individual publications would obviously have to be fine tuned, and could be handled by computers. At the end of the day, the specific ways that manuscripts are graded could become a hallmark of individual journals, entering in the choice that authors make between one or another publisher when submitting their data for publication. But authors would be the payers, and would therefore naturally be in the position of choosers, rather than the beggars as they too often end up being nowadays.

8) The papers would therefore be awarded scores, and graded in accordance. A possible grading system would be as follows:

- The bottom 50%, that had been refereed to ensure of their scientific soundness would remain graded as Sound.
- The next 25% (from 50 to 75 %) would be called Interesting
- The next 15% (from 75 to 90 %) would be called Very Interesting
- The next 9% would be called Outstanding
- The top 1% would be called Exceptional

Such a grading system has in fact already been developed by the 'Faculty of 1000', and seems to work very well to help scientists recognise the papers that are worth downloading and reading. The difference in the system proposed here is that the evaluation would primarily be done on the scientific impact, evaluated through citations of the paper itself, and not based on the opinion of a handful of established scientists, who may be resistant to truly novel ideas.

Even if the publication costs effectively paid by the authors should be related to the impact of the paper, I do not believe that the publishers should wait for the end of the six months period to charge the authors. Rather, I feel that all authors should be charged the same amount from the start, at the time of publication. In this scheme, scientific publishers are there to make money, and compete with one another for the business by offering authors the best possible service. In such a business-driven world, delaying payment by six months would not be a small drawback. After six months, however, the authors of papers that have been graded better than simply sound would receive a redemption voucher, to be used against payment for subsequent papers to be published in that journal (or by the same publisher).

As far as the rates being charged, I would envisage something along the following lines:
The authors would initially be charged a set fee corresponding to twice the actual cost of publishing their paper.
Interesting papers would be published at cost (and therefore get half their money back in the form of a voucher)
Very Interesting papers would be charged half the cost (3/4 money back).
Oustanding papers would be published for free, and Exceptional papers would also be published for free, but the authors would also receive an award of let's say US$1000 (in real money, not vouchers this time).

According to my simple calculations, the global return of such a system for the publisher would be 132 % of the running costs, which seems more than adequate for them to make a viable profit, and therefore for competition among publishers to ensure of the quality of the services rendered (and value for money).

I even think that the top-rated journals, such as Science, Nature and Cell (and their numerous successful clones), could very well evolve profitably towards a similar system. Personally, I find that I read the News and Views section of Nature much more often than the actual papers themselves. And I feel that this is not a good situation for at least two reasons: First, because I end up reading something written by somebody other than the person who actually did the work. And second, when work published in Nature is so relevant to my field of work that I really read the papers themselves, I often feel frustrated by how trimmed and skinned these papers are. The 'Materials and Methods' provided are usually so scant that you could not possibly reproduce experiments from that information. And the need to "make the news" often detracts from careful scientific reasoning. If the publishers of these 'top' journals were to adopt a model following that described above, one could imagine that they would then select those upper 10% or upper 1% of papers and invite authors to write a synthetic news piece about their work, to be published in the paper magazine in parallel to the original online scientific paper. It is also true that some authors are not as good at writing as other well-spoken gurus in certain fields. But it would be possible to set up a joint authorship system, or a "pas de deux", where authors would write the initial summary and a third party commentator would emphasise the relevance of the work and put it into bigger perspective at a second stage. But at least the people who actually did the work would be writing in the section that is read by the broadest audience. The paper issues of Nature and Science would then become true scientific magazines, the ones you can take to the coffee room, or to more solitary places, to browse at your leisure. And the publishers would have double incomes, from the hundreds of manuscripts published on their website on the one hand, and from subscriptions to their newsy magazines on the other.

To conclude, I would say that, if such a system were to replace the current practice of scientific publishing, there would be an even bigger advantage which I have not alluded to yet: there would probably be fewer papers published, but the average quality of individual papers would be increased.

At present, there is an enormous pressure for people to publish numerous papers. But if authors paid relatively high prices to publish low-key papers, and if there was effectively no limit to how many papers you could publish, the number of papers published would no longer be a way of assessing productivity. The emphasis for this would presumably evolve towards considering how many interesting papers a researcher has published. And so scientists would have to spend more time ensuring the quality of the science they publish, and less time trying to publish papers based on the minimal amount of data possible.

As somebody who tries to keep abreast of his field by spending several hours per week scanning dozens of tables of contents, I would certainly greatly appreciate if there were fewer papers published, but if the average content of those was more substantial and of higher scientific quality.