This article is part of the supplement: Machine Learning for Biomedical Literature Analysis and Text Retrieval

Open Access Open Badges Research

A system for de-identifying medical message board text

Adrian Benton1*, Shawndra Hill2, Lyle Ungar3, Annie Chung1, Charles Leonard1, Cristin Freeman1 and John H Holmes1

Author Affiliations

1 University of Pennsylvania School of Medicine, Philadelphia, PA

2 University of Pennsylvania, The Wharton School, Philadelphia, PA

3 University of Pennsylvania School of Engineering and Applied Science, Philadelphia, PA

For all author emails, please log on.

BMC Bioinformatics 2011, 12(Suppl 3):S2  doi:10.1186/1471-2105-12-S3-S2

Published: 9 June 2011


There are millions of public posts to medical message boards by users seeking support and information on a wide range of medical conditions. It has been shown that these posts can be used to gain a greater understanding of patients’ experiences and concerns. As investigators continue to explore large corpora of medical discussion board data for research purposes, protecting the privacy of the members of these online communities becomes an important challenge that needs to be met. Extant entity recognition methods used for more structured text are not sufficient because message posts present additional challenges: the posts contain many typographical errors, larger variety of possible names, terms and abbreviations specific to Internet posts or a particular message board, and mentions of the authors’ personal lives. The main contribution of this paper is a system to de-identify the authors of message board posts automatically, taking into account the aforementioned challenges. We demonstrate our system on two different message board corpora, one on breast cancer and another on arthritis. We show that our approach significantly outperforms other publicly available named entity recognition and de-identification systems, which have been tuned for more structured text like operative reports, pathology reports, discharge summaries, or newswire.