Research article

Various criteria in the evaluation of biomedical named entity recognition

Richard Tzong-Han Tsai^1,2 , Shih-Hung Wu^1,3 , Wen-Chi Chou¹ , Yu-Chun Lin^1,2 , Ding He¹ , Jieh Hsiang² , Ting-Yi Sung¹ and Wen-Lian Hsu¹

¹  Institute of Information Science, Academia Sinica, Nankang, Taipei 115, R.O.C, Taiwan

²  Department of Computer Science and Information Engineering, National Taiwan University, Taipei 106, R.O.C, Taiwan

³  Department of Computer Science and Information Engineering, Chaoyang University of Technology, Taichung County 413, R.O.C, Taiwan

author email corresponding author email

BMC Bioinformatics 2006, 7:92doi:10.1186/1471-2105-7-92

Published:	24 February 2006

Abstract

Background

Text mining in the biomedical domain is receiving increasing attention. A key component of this process is named entity recognition (NER). Generally speaking, two annotated corpora, GENIA and GENETAG, are most frequently used for training and testing biomedical named entity recognition (Bio-NER) systems. JNLPBA and BioCreAtIvE are two major Bio-NER tasks using these corpora. Both tasks take different approaches to corpus annotation and use different matching criteria to evaluate system performance. This paper details these differences and describes alternative criteria. We then examine the impact of different criteria and annotation schemes on system performance by retesting systems participated in the above two tasks.

Results

To analyze the difference between JNLPBA's and BioCreAtIvE's evaluation, we conduct Experiment 1 to evaluate the top four JNLPBA systems using BioCreAtIvE's classification scheme. We then compare them with the top four BioCreAtIvE systems. Among them, three systems participated in both tasks, and each has an F-score lower on JNLPBA than on BioCreAtIvE. In Experiment 2, we apply hypothesis testing and correlation coefficient to find alternatives to BioCreAtIvE's evaluation scheme. It shows that right-match and left-match criteria have no significant difference with BioCreAtIvE. In Experiment 3, we propose a customized relaxed-match criterion that uses right match and merges JNLPBA's five NE classes into two, which achieves an F-score of 81.5%. In Experiment 4, we evaluate a range of five matching criteria from loose to strict on the top JNLPBA system and examine the percentage of false negatives. Our experiment gives the relative change in precision, recall and F-score as matching criteria are relaxed.

Conclusion

In many applications, biomedical NEs could have several acceptable tags, which might just differ in their left or right boundaries. However, most corpora annotate only one of them. In our experiment, we found that right match and left match can be appropriate alternatives to JNLPBA and BioCreAtIvE's matching criteria. In addition, our relaxed-match criterion demonstrates that users can define their own relaxed criteria that correspond more realistically to their application requirements.