Open Access Highly Accessed Database

A benchmark server using high resolution protein structure data, and benchmark results for membrane helix predictions

Emma M Rath1, Dominique Tessier2, Alexander A Campbell1, Hong Ching Lee13, Tim Werner1, Noeris K Salam14, Lawrence K Lee15 and W Bret Church1*

Author Affiliations

1 Group in Biomolecular Structure and Informatics, Faculty of Pharmacy, The University of Sydney, Darlinghurst, Sydney, NSW 2006, Australia

2 UR 1268 Biopolymères Interactions Assemblages, INRA, Nantes, 44300, France

3 Garvan Institute of Medical Research, Darlinghurst, Sydney, NSW 2010, Australia

4 Current address: Schrödinger Inc., 8910 University Center Lane, San Diego, CA, 92122, USA

5 Current address: Victor Chang Cardiac Research Institute, Lowy Packer Building, 405 Liverpool St, Darlinghurst, NSW 2010, Australia

For all author emails, please log on.

BMC Bioinformatics 2013, 14:111  doi:10.1186/1471-2105-14-111

Published: 27 March 2013



Helical membrane proteins are vital for the interaction of cells with their environment. Predicting the location of membrane helices in protein amino acid sequences provides substantial understanding of their structure and function and identifies membrane proteins in sequenced genomes. Currently there is no comprehensive benchmark tool for evaluating prediction methods, and there is no publication comparing all available prediction tools. Current benchmark literature is outdated, as recently determined membrane protein structures are not included. Current literature is also limited to global assessments, as specialised benchmarks for predicting specific classes of membrane proteins were not previously carried out.


We present a benchmark server at webcite that uses recent high resolution protein structural data to provide a comprehensive assessment of the accuracy of existing membrane helix prediction methods. The server further allows a user to compare uploaded predictions generated by novel methods, permitting the comparison of these novel methods against all existing methods compared by the server. Benchmark metrics include sensitivity and specificity of predictions for membrane helix location and orientation, and many others. The server allows for customised evaluations such as assessing prediction method performances for specific helical membrane protein subtypes.

We report results for custom benchmarks which illustrate how the server may be used for specialised benchmarks. Which prediction method is the best performing method depends on which measure is being benchmarked. The OCTOPUS membrane helix prediction method is consistently one of the highest performing methods across all measures in the benchmarks that we performed.


The benchmark server allows general and specialised assessment of existing and novel membrane helix prediction methods. Users can employ this benchmark server to determine the most suitable method for the type of prediction the user needs to perform, be it general whole-genome annotation or the prediction of specific types of helical membrane protein. Creators of novel prediction methods can use this benchmark server to evaluate the performance of their new methods. The benchmark server will be a valuable tool for researchers seeking to extract more sophisticated information from the large and growing protein sequence databases.

Helical membrane proteins; Transmembrane helix prediction; Benchmark