Email updates

Keep up to date with the latest news and content from BMC Medical Research Methodology and BioMed Central.

Open Access Research article

Evaluating current automatic de-identification methods with Veteran’s health administration clinical documents

Oscar Ferrández12*, Brett R South12, Shuying Shen12, F Jeffrey. Friedlin3, Matthew H Samore12 and Stéphane M Meystre12

Author Affiliations

1 Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA

2 IDEAS Center SLCVA Healthcare System, Salt Lake City, UT, USA

3 Medical Informatics, Regenstrief Institute, Inc, Indianapolis, IN, USA

For all author emails, please log on.

BMC Medical Research Methodology 2012, 12:109  doi:10.1186/1471-2288-12-109

Published: 27 July 2012

Abstract

Background

The increased use and adoption of Electronic Health Records (EHR) causes a tremendous growth in digital information useful for clinicians, researchers and many other operational purposes. However, this information is rich in Protected Health Information (PHI), which severely restricts its access and possible uses. A number of investigators have developed methods for automatically de-identifying EHR documents by removing PHI, as specified in the Health Insurance Portability and Accountability Act “Safe Harbor” method.

This study focuses on the evaluation of existing automated text de-identification methods and tools, as applied to Veterans Health Administration (VHA) clinical documents, to assess which methods perform better with each category of PHI found in our clinical notes; and when new methods are needed to improve performance.

Methods

We installed and evaluated five text de-identification systems “out-of-the-box” using a corpus of VHA clinical documents. The systems based on machine learning methods were trained with the 2006 i2b2 de-identification corpora and evaluated with our VHA corpus, and also evaluated with a ten-fold cross-validation experiment using our VHA corpus. We counted exact, partial, and fully contained matches with reference annotations, considering each PHI type separately, or only one unique ‘PHI’ category. Performance of the systems was assessed using recall (equivalent to sensitivity) and precision (equivalent to positive predictive value) metrics, as well as the F2-measure.

Results

Overall, systems based on rules and pattern matching achieved better recall, and precision was always better with systems based on machine learning approaches. The highest “out-of-the-box” F2-measure was 67% for partial matches; the best precision and recall were 95% and 78%, respectively. Finally, the ten-fold cross validation experiment allowed for an increase of the F2-measure to 79% with partial matches.

Conclusions

The “out-of-the-box” evaluation of text de-identification systems provided us with compelling insight about the best methods for de-identification of VHA clinical documents. The errors analysis demonstrated an important need for customization to PHI formats specific to VHA documents. This study informed the planning and development of a “best-of-breed” automatic de-identification application for VHA clinical text.

Keywords:
Confidentiality, patient data privacy [MeSH F04.096.544.335.240]; Natural language processing [L01.224.065.580]; Health insurance portability and accountability act [N03.219.521.576.343.349]; De-identification; Anonymization; Electronic health records [E05.318.308.940.968.625.500]; United States department of veterans affairs [I01.409.137.500.700]