This article is part of the supplement: Proceedings of the 2009 AMIA Summit on Translational Bioinformatics

Open Access Open Badges Proceedings

Developing a manually annotated clinical document corpus to identify phenotypic information for inflammatory bowel disease

Brett R South123*, Shuying Shen123, Makoto Jones2, Jennifer Garvin12, Matthew H Samore123, Wendy W Chapman4 and Adi V Gundlapalli123*

Author Affiliations

1 VA Salt Lake City Health Care System, IDEAS Center, Salt Lake City, Utah, 84148, USA

2 Department of Internal Medicine, University of Utah, Division of Clinical Epidemiology, Salt Lake City, Utah, 84148, USA

3 Department of Biomedical Informatics, University of Utah, Salt Lake City, Utah, 84148, USA

4 Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, PA, USA

For all author emails, please log on.

BMC Bioinformatics 2009, 10(Suppl 9):S12  doi:10.1186/1471-2105-10-S9-S12

Published: 17 September 2009



Natural Language Processing (NLP) systems can be used for specific Information Extraction (IE) tasks such as extracting phenotypic data from the electronic medical record (EMR). These data are useful for translational research and are often found only in free text clinical notes. A key required step for IE is the manual annotation of clinical corpora and the creation of a reference standard for (1) training and validation tasks and (2) to focus and clarify NLP system requirements. These tasks are time consuming, expensive, and require considerable effort on the part of human reviewers.


Using a set of clinical documents from the VA EMR for a particular use case of interest we identify specific challenges and present several opportunities for annotation tasks. We demonstrate specific methods using an open source annotation tool, a customized annotation schema, and a corpus of clinical documents for patients known to have a diagnosis of Inflammatory Bowel Disease (IBD). We report clinician annotator agreement at the document, concept, and concept attribute level. We estimate concept yield in terms of annotated concepts within specific note sections and document types.


Annotator agreement at the document level for documents that contained concepts of interest for IBD using estimated Kappa statistic (95% CI) was very high at 0.87 (0.82, 0.93). At the concept level, F-measure ranged from 0.61 to 0.83. However, agreement varied greatly at the specific concept attribute level. For this particular use case (IBD), clinical documents producing the highest concept yield per document included GI clinic notes and primary care notes. Within the various types of notes, the highest concept yield was in sections representing patient assessment and history of presenting illness. Ancillary service documents and family history and plan note sections produced the lowest concept yield.


Challenges include defining and building appropriate annotation schemas, adequately training clinician annotators, and determining the appropriate level of information to be annotated. Opportunities include narrowing the focus of information extraction to use case specific note types and sections, especially in cases where NLP systems will be used to extract information from large repositories of electronic clinical note documents.