This article is part of the supplement: The Third BioCreative – Critical Assessment of Information Extraction in Biology Challenge

Open Access Research

Benchmarking of the 2010 BioCreative Challenge III text-mining competition by the BioGRID and MINT interaction databases

Andrew Chatr-aryamontri1*, Andrew Winter1, Livia Perfetto2, Leonardo Briganti2, Luana Licata2, Marta Iannuccelli2, Luisa Castagnoli2, Gianni Cesareni23* and Mike Tyers14*

Author Affiliations

1 School of Biological Sciences, University of Edinburgh, Edinburgh EH9 3JR, UK

2 Department of Biology, University of Rome Tor Vergata, Rome 00133, Italy

3 IRCSS, Fondazione Santa Lucia, Rome 00143, Italy

4 Center for Systems Biology, Samuel Lunenfeld Research Institute, Mount Sinai Hospital, Toronto, Canada

For all author emails, please log on.

BMC Bioinformatics 2011, 12(Suppl 8):S8  doi:10.1186/1471-2105-12-S8-S8

Published: 3 October 2011



The vast amount of data published in the primary biomedical literature represents a challenge for the automated extraction and codification of individual data elements. Biological databases that rely solely on manual extraction by expert curators are unable to comprehensively annotate the information dispersed across the entire biomedical literature. The development of efficient tools based on natural language processing (NLP) systems is essential for the selection of relevant publications, identification of data attributes and partially automated annotation. One of the tasks of the Biocreative 2010 Challenge III was devoted to the evaluation of NLP systems developed to identify articles for curation and extraction of protein-protein interaction (PPI) data.


The Biocreative 2010 competition addressed three tasks: gene normalization, article classification and interaction method identification. The BioGRID and MINT protein interaction databases both participated in the generation of the test publication set for gene normalization, annotated the development and test sets for article classification, and curated the test set for interaction method classification. These test datasets served as a gold standard for the evaluation of data extraction algorithms.


The development of efficient tools for extraction of PPI data is a necessary step to achieve full curation of the biomedical literature. NLP systems can in the first instance facilitate expert curation by refining the list of candidate publications that contain PPI data; more ambitiously, NLP approaches may be able to directly extract relevant information from full-text articles for rapid inspection by expert curators. Close collaboration between biological databases and NLP systems developers will continue to facilitate the long-term objectives of both disciplines.