Email updates

Keep up to date with the latest news and content from BMC Bioinformatics and BioMed Central.

Open Access Research article

Dataset size and composition impact the reliability of performance benchmarks for peptide-MHC binding predictions

Yohan Kim1, John Sidney1, Søren Buus2, Alessandro Sette1, Morten Nielsen34 and Bjoern Peters1*

Author Affiliations

1 La Jolla Institute for Allergy & Immunology, 9420 Athena Circle, La Jolla, CA 92037, USA

2 Department of International Health, Immunology and Microbiology, University of Copenhagen, Blegdamsvej 3, 2200 København N, Copenhagen, Denmark

3 Center for Biological Sequence Analysis, Department of Systems Biology, The Technical University of Denmark, Building 208, Lyngby DK-2800, Denmark

4 Instituto de Investigaciones Biotecnológicas, Universidad Nacional de San Martín, San Martín, Buenos Aires B 1650 HMP, Argentina

For all author emails, please log on.

BMC Bioinformatics 2014, 15:241  doi:10.1186/1471-2105-15-241

Published: 14 July 2014

Abstract

Background

It is important to accurately determine the performance of peptide:MHC binding predictions, as this enables users to compare and choose between different prediction methods and provides estimates of the expected error rate. Two common approaches to determine prediction performance are cross-validation, in which all available data are iteratively split into training and testing data, and the use of blind sets generated separately from the data used to construct the predictive method. In the present study, we have compared cross-validated prediction performances generated on our last benchmark dataset from 2009 with prediction performances generated on data subsequently added to the Immune Epitope Database (IEDB) which served as a blind set.

Results

We found that cross-validated performances systematically overestimated performance on the blind set. This was found not to be due to the presence of similar peptides in the cross-validation dataset. Rather, we found that small size and low sequence/affinity diversity of either training or blind datasets were associated with large differences in cross-validated vs. blind prediction performances. We use these findings to derive quantitative rules of how large and diverse datasets need to be to provide generalizable performance estimates.

Conclusion

It has long been known that cross-validated prediction performance estimates often overestimate performance on independently generated blind set data. We here identify and quantify the specific factors contributing to this effect for MHC-I binding predictions. An increasing number of peptides for which MHC binding affinities are measured experimentally have been selected based on binding predictions and thus are less diverse than historic datasets sampling the entire sequence and affinity space, making them more difficult benchmark data sets. This has to be taken into account when comparing performance metrics between different benchmarks, and when deriving error estimates for predictions based on benchmark performance.

Keywords:
Benchmarking of MHC class I predictors; Epitope prediction; Sequence similarity; Cross-validation