 Freedom of Information Conference 2000
Barry Markovitz
Washington University
Medical School
What's wrong with how we judge science?
We
must learn to value the work, not the place of publication
It is a bright
sunny day and you are walking along a bustling downtown street doing some
window shopping. You pass a trendy department store, known worldwide for
its high quality merchandise and cutting edge apparel. You see strange
new fashions draped on the stark window models and think, almost subconsciously,
that this is quality, this is "happening," because it is in
this particular store's window.
You return
to your office and pick up a copy of the Lancet. You turn immediately
to an original research article on a subject you are interested in. Though
you know you should read the whole study critically, you have patients
waiting to be seen and you jump to the conclusions, confident that the
study must be of excellent quality because it has appeared in such a prestigious
and high impact journal.
Children
in elementary school have a useful admonition to guard against what you
have just done, in both circumstances: "don't judge a book by its
cover." This is, in the opinion of some, the primary shortcoming
of the way the community of biomedicine judges the quality of science
and scientific publication.
Scientific
research is made public to disseminate knowledge, influence future investigation,
and establish the academic credentials of the scientist. The print, peer
reviewed biomedical journal has been the "gold standard" publication
tool for centuries, but it is being challenged on multiple fronts as too
expensive, too inefficient, too limited, and perhaps even superfluous
in our web enabled society. Indeed, the model whereby research is funded
largely by the public, scientists turn over their intellectual product
freely to a commercial third party, and are then forced to buy it back
at a premium price is failing fast. New ways to retain the quality control
and filtering features of editing and peer review are being considered,
while using the internet as the primary distribution medium is truly making
publication public.
But what
is quality in scientific publication? How is it judged? Where have we
failed, and can new models of open access to the biomedical literature
offer a better way?
Citation
analysis flawed
Reasoning that the impact of a scientific paper on subsequent research,
as judged by the number of times a paper is cited by others, is representative
of its innate quality, citation analysis can be a useful measurement tool.
On average, all things being equal, there is little doubt that the more
important the study, the more likely it is to be cited. For several decades,
the Institute of Scientific Information (ISI) in Philadelphia, founded
by Eugene Garfield, has tracked thousands of journals and the millions
of citations between them.
But there
are flaws in the ISI's Science Citation Index.[1] The set of journals
covered is incomplete and varies between fields of study. There is an
English language and a United States publication bias. There are technical
errors including misspellings, synonymy (several variations of the same
article), and homonymy (several authors with the same name). However,
perhaps the greatest fallacy rests with the motivations and factors that
influence the creation of citation lists in individual papers in the first
place. Indeed, upon closer inspection, the primary criterion for including
a reference in a citation list is its utility in support of a manuscript,
not its quality. Review articles are often cited in lieu of primary studies.
Citations can be negative, argumentative, or simply included to flatter
editors or peer reviewers (by citing the editor or potential peer reviewers,
or by citing "hot papers" to show off). The biggest hurdle to
using citation analysis in the evaluation of publications is the cost
to access ISI's database and time needed for a thorough analysis. (The
new method of access, via ISI's Web of Science, does offer a simple and
efficient interface, though it remains expensive.)
The dangers
of using journal impact factors
A far too convenient short cut has evolved. Rather than assess the citation
rate of an individual paper or author, why not use the average citation
rate of the journal in which the report appears? The ISI has facilitated
this short cut by publishing journal impact factors for several thousand
journals. The impact factor is calculated as the number of citations a
journal receives in a given year to articles from the previous two years,
divided by total published (source) articles in the previous two years.
Increasingly, in evaluating faculty for promotion and investigators for
funding, where a scientist has published is more important that what they
have published. The assumption is made that publication in a high impact
journal implies that the paper itself will or has had a high impact, and
is therefore of high quality. Eugene Garfield himself has warned against
this practice: "Using the journal's average citation impact instead
of the actual article's impact is tantamount to grading by the prestige
of the journal involved. While expedient, it is dangerous."[2]
It is dangerous
because citation patterns vary tremendously between fields of inquiry,
so that many specialty areas have no high impact journals as defined by
the usual values. Does this imply that all the journals in this field--and
therefore all the papers--are of low quality? It is dangerous because
it essentially assigns an average citation rate to all the papers in the
journal, when in fact the distribution of citedness of articles, even
in high impact journals, is far from a normal distribution (figure 1).
Indeed, it has been estimated that 50% of a journal's citations can be
accounted for by 15% of its papers.[1] It is dangerous because it assumes
a correlation between the citation rate of an individual author's papers
over time and the impact factors of the journals he or she has published
in, and this correlation does not exist (figure 2).
figure
1
figure
2
The prospects
of open access
Open access to the world's biomedical literature offers tremendous potential
for testing new methods to assess publication quality. The restrained
and distinguished tools of bibliometrics have morphed into webometrics
and cybermetrics. One can now track not only citation rates (which could
potentially increase dramatically with the increased exposure that open
access can afford) but actual usage or download rates. New search engines
can determine the number of links on the web to a particular URL, a potentially
dynamic and more comprehensive assessment of how important a document
is in the eyes of the world at large.
Of course
there are potential pitfalls along the way. We must be on guard against
the surreptitious self citation equivalent of downloading one's own papers,
or engaging in covert arrangement schemes of linking to each others' papers.
We will need to learn how to interpret the "Slashdot effect,"
where interest in a web resource surges after its announcement on a public
news server.[3]
The major
quality control filter that currently exists in biomedical and scholarly
publishing today is the peer review and editing process. Imperfect though
it may be, peer review does tend to function as an admissions committee
of sorts. Indeed, the most prestigious medical journals boast of high
rejection rates. A manuscript that has been rejected by several journals
is unlikely to be of very high quality. Can the peer review and editing
process be made rigorous enough or sufficiently systematic to mark or
rank a paper on a quality scale? Unfortunately, as Peter Singer explains
(see article by Peter Singer) we are discussing quality in a vacuum, for
this is a measure without a gold standard. Singer suggests that quality
should be measured within a system that rewards authors who retain copyright
and ensure that their work is available to their peers without access
restrictions.
Ultimately
all facets of our current biomedical publishing model, with its Balkanization
of science into closed, often illogical journal fiefdoms, unscientific
methods of assigning quality, obtuse reward system, and heavy tolls required
for access must be considered together as we lurch inevitably towards
a future where, as Stevan Harnad envisions: "all papers in all fields
[are] systematically interconnected, effortlessly accessible and rationally
navigable, from any researcher's desk, worldwide for free."[4]
References
- Seglen PO
Why the impact factor of journals should not be used for evaluating research.
Br Med J 1997;314:497. Available at: http://www.bmj.com/cgi/content-nw/full/314/7079/497
- Garfield E. Fortnightly Review: How can impact factors be improved? Br
Med J 1996; 313: 411-413. Available at: http://www.bmj.com/cgi/content/full/313/7054/411
- Adler S. The Slashdot Effect: An Analysis of Three Internet Publications.
Available at: http://ssadler.phy.bnl.gov/adler/SDE/SlashDotEffect.html
Accessed July 13, 2000.
- Harnad S. Integrating and Navigating Eprint Archives Through Citation-Linking.
Available at: http://www.cogsci.soton.ac.uk/~harnad/citation.html
Accessed July 13, 2000.
Barry P Markovitz
Washington University School of Medicine
St. Louis, Missouri, USA
Competing
interest: None declared.
|
