Email updates

Keep up to date with the latest news and content from BMC Bioinformatics and BioMed Central.

This article is part of the supplement: Proceedings of the 11th Annual Bioinformatics Open Source Conference (BOSC) 2010

Open Access Proceedings

Hybrid cloud and cluster computing paradigms for life science applications

Judy Qiu12*, Jaliya Ekanayake12, Thilina Gunarathne12, Jong Youl Choi12, Seung-Hee Bae12, Hui Li12, Bingjing Zhang12, Tak-Lon Wu12, Yang Ruan12, Saliya Ekanayake12, Adam Hughes12 and Geoffrey Fox12

Author affiliations

1 School of Informatics and Computing, Indiana University, Bloomington, IN 47405, USA

2 Pervasive Technology Institute, Indiana University, Bloomington, IN 47408, USA

For all author emails, please log on.

Citation and License

BMC Bioinformatics 2010, 11(Suppl 12):S3  doi:10.1186/1471-2105-11-S12-S3

Published: 21 December 2010

Abstract

Background

Clouds and MapReduce have shown themselves to be a broadly useful approach to scientific computing especially for parallel data intensive applications. However they have limited applicability to some areas such as data mining because MapReduce has poor performance on problems with an iterative structure present in the linear algebra that underlies much data analysis. Such problems can be run efficiently on clusters using MPI leading to a hybrid cloud and cluster environment. This motivates the design and implementation of an open source Iterative MapReduce system Twister.

Results

Comparisons of Amazon, Azure, and traditional Linux and Windows environments on common applications have shown encouraging performance and usability comparisons in several important non iterative cases. These are linked to MPI applications for final stages of the data analysis. Further we have released the open source Twister Iterative MapReduce and benchmarked it against basic MapReduce (Hadoop) and MPI in information retrieval and life sciences applications.

Conclusions

The hybrid cloud (MapReduce) and cluster (MPI) approach offers an attractive production environment while Twister promises a uniform programming environment for many Life Sciences applications.

Methods

We used commercial clouds Amazon and Azure and the NSF resource FutureGrid to perform detailed comparisons and evaluations of different approaches to data intensive computing. Several applications were developed in MPI, MapReduce and Twister in these different environments.