To evaluate the performance of CGP methods, three case studies were selected for rediscovery experiments using well-known pathway genes as gold-standards. For each case study, the polypeptide sequences of n candidate genes were compared with all open reading frames (orf) of the k genome sequences from the National Centre for Biotechnology Information (NCBI, accessed April 2007) by Basic Local Alignment and Search Tool (BLASTP) http://blast.ncbi.nlm.nih.gov/Blast.cgi. If a candidate gene reached the critical E-value of < 10^-5 in a given genome, a gene or gene homolog was defined as present in the genome. If a gene did not reach the critical E-value in a genome, the gene was recorded as absent from the genome. The binary states of gene occurrence were recorded in an n × k homolog matrix.

From the k genomes, k_p genomes known to display the phenotype of interest (p) were selected as positive genome examples, and k_n genomes not displaying p were chosen as negative genome examples. For each of the n candidate genes, the number of co-presence (homologs present in positive genome examples) and co-absence (homologs absent in negative genome examples) were counted and presented into a 2 × 2 contingency table, from which a number of statistical scoring functions was calculated. The scoring functions included: a) sensitivity (sens, the proportion of genes present in the positive genome examples), b) specificity (spec, proportion of genes absent in the negative genome examples), c) positive and negative predictive values (ppv/npv, the proportion of positive/negative genomes were present/absent when the gene was present/absent), d) arithmetic (amss) and harmonic (hmss) mean of sensitivity and specificity, e) odds ratio (OR, the odds of a gene existed in the positive example versus the odds of a gene was absent in the negative examples), f) chi-square scoring function (chisq, the deviation of the observed frequency from the expected proportion), g) directional chi-square function (bchisq, the chisq function with genes that displayed inverse associations be reversed to the bottom of the rank), and h) F-measure (F, the harmonic mean of the sensitivity and precision). The mathematical definitions of these scoring functions are listed in the Additionl file 1.

Inductive CGP ranks genes by finding genes with similar occurrence pattern across a number of bacterial genomes using supervised machine learning. A number of genes known to display a target phenotype or function p were selected as positive examples for the training set. Similarly, genes that did not contribute to p were selected as negative gene examples. The occurrences of genes in k genome examples were used as features for model training. Candidate genes were ranked by the score or posterior probability from the output of the machine learning classifiers. The machine learning classifiers included naïve Bayes (NB), logistic regression (LR; ridge = 10^-5), J48 decision tree (J48, pruning confidence = 0.25), nearest neighbour classifier (IBk, with inverse distance weighing; k was determined by leave-one-out cross-validation), alternating decision tree (ADTree; boosting iteration = 10), support vector machines (SVM) with polynomial (SVM/Poly; linear kernel trained by sequential minimal optimisation algorithm, SMO) and radial basis function (SVM/RBF; trained by SMO; γ = 0.01) kernels. The Waikato Environment for Knowledge Analysis (WEKA) 3.5.6 was used for classifier training 21. For the purpose of benchmarking, the generalisation performance of inductive CGP was evaluated by stratified 10-fold cross-validation: for the n genes used as candidate genes for prioritisation, all n₊ genes from the validation set and the rest of n_- genes not in the validation set were each randomly divided into 10 subsets. One-tenth of the the genes from each group (110 MathType@MTEF@1@1@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaqcfa4aaSaaaeaacqaIXaqmaeaacqaIXaqmcqaIWaamaaaaaa@2F41@ of n₊ and n_- genes) were sequentially selected as test set, whereas the rest of the genes were selected as training set to train inductive models. The performance of each inductive CGP algorithm was obtained by averaging areas under receiver operating characteristic curves (AUC) over the 10 runs.

The performance of different CGP methods was evaluated by rediscovery experiments. The relative position of the ranked candidate gene was measured by percentiles from the top of the rank (pct). The AUCs were estimated non-parametrically by trapezoidal rule. We adopted probability enrichment (the relative enrichment ratio) described by Turner et al 7 to compare the performance of different statistical CGP scoring functions [see Additional file 1]. The average and maximum probability enrichments, defined as n folds-improvement in precision above a certain score threshold τ, were calculated by partial precision (pppv), such that:

Partial prec .  at  τ = p p p v ( τ ) = Num .  correct genes > τ Num .  genes > τ MathType@MTEF@1@1@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaqbaeaabiWaaaqaaiabbcfaqjabbggaHjabbkhaYjabbsha0jabbMgaPjabbggaHjabbYgaSjabbccaGiabbchaWjabbkhaYjabbwgaLjabbogaJjabc6caUiabbccaGiabbggaHjabbsha0jabbccaGiabes8a0bqaaiabg2da9aqaaiabdchaWjabdchaWjabdchaWjabdAha2jabcIcaOiabes8a0jabcMcaPaqaaaqaaiabg2da9aqcfayaamaalaaabaGaeeOta4KaeeyDauNaeeyBa0MaeiOla4IaeeiiaaIaee4yamMaee4Ba8MaeeOCaiNaeeOCaiNaeeyzauMaee4yamMaeeiDaqNaeeiiaaIaee4zaCMaeeyzauMaeeOBa4MaeeyzauMaee4CamNaeyOpa4JaeqiXdqhabaGaeeOta4KaeeyDauNaeeyBa0MaeiOla4IaeeiiaaIaee4zaCMaeeyzauMaeeOBa4MaeeyzauMaee4CamNaeyOpa4JaeqiXdqhaaaaaaaa@773B@

and the average (η¯ MathType@MTEF@1@1@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGafq4TdGMbaebaaaa@2D99@) and the maximum (η_max) probability enrichments were defined as:

η ( t ) = p p p v n ( t ) p p v η ¯ = ∫ 0 1 η ( t ) d t η m a x = η ( t ∗ ) MathType@MTEF@1@1@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaqbaeWabmqaaaqaaiabeE7aOjabcIcaOiabdsha0jabcMcaPiabg2da9KqbaoaalaaabaGaemiCaaNaemiCaaNaemiCaaNaemODay3aaSbaaeaacqWGUbGBaeqaaiabcIcaOiabdsha0jabcMcaPaqaaiabdchaWjabdchaWjabdAha2baaaOqaaiqbeE7aOzaaraGaeyypa0Zaa8qmaeaacqaH3oaAcqGGOaakcqWG0baDcqGGPaqkcqWGKbazcqWG0baDaSqaaiabicdaWaqaaiabigdaXaqdcqGHRiI8aaGcbaGaeq4TdG2aaSbaaSqaaiabd2gaTjabdggaHjabdIha4bqabaGccqGH9aqpcqaH3oaAcqGGOaakcqWG0baDdaahaaWcbeqaaiabgEHiQaaakiabcMcaPaaaaaa@5CC1@

where t was the rank fraction at threshold τ, ppv was the overall precision, pppv_n(t) was the partial precision at rank fraction t, and η(t) was at its maximum at t*. Both AUC and measure the overall performance of a CGP task. The rank fraction t* indicates the point above which correct genes are likely to be found η_max-times more likely than compared to a random gene list. Evaluation with η_max is useful to identify cases where a small proportion of genes is ranked highly but the overall performance is poor.

Two simulation experiments were performed to investigate the effect of the number of genome examples on statistical CGP performance (Case study 1, see below). For the first simulation, the amss scoring function was repeatedly applied on randomly selected subsets of 417 positive and negative genome examples, using genes from set M (Case study 1). The number of positive genome examples (N_p) and negative genome examples (N_n) were gradually increased in each subset. For each combination of N_p and N_n, 25 runs were performed and the median AUC was obtained. A second simulation was performed to determine the variability of performance. Here the proportion of positive and negative genome examples was kept the same (400:17) and the median and the range of AUC were then obtained over 1000 runs for each N_p and N_n.

A similar simulation was also performed to determine the effect of genome example sizes on inductive CGP performance. Twenty five subsets of N genomes (from 417 genomes) were randomly selected as features with N increased from 1 to 417. For each N, stratified 10-fold cross-validations were performed with SVM/Poly using all genes from SA-2603 genome as candidates. Median AUCs from 25 random subset of N genomes were obtained.

The best scoring functions for rediscovering metabolic genes (M set) were the amss, hmss, and npv (AUC >0.970) using the whole genome of SA-2603 (2124 genes) as candidate genes. Of the 25 known peptidoglycan-related genes, all except one gene were identified within the top 13% (median: top 1.0 pct) in SA-2603 (Table 1 and Figure 1). The top-scored genes in the SA-2603 genome are listed in [see Additional file 7]. Encouraging results were also achieved in prioritising the EC-K12 genes in all three validation sets (Table 2); for example, for the M set genes in EC-K12 (51 known genes out of 4134 genes in the bacterial genome), an AUC of 0.911 was achieved by amss, and the median of the rediscovered genes was at the top 3.2 pct of the rank. In contrast, poor performances were yielded when matching the control validation sets (glycolysis) against the same amss-prioritised ranks (SA-2603: 0.398; EC-K12: 0.341; Figure 1).

The performance of statistical CGP was also measured by folds-increase in precision (probability enrichments) compared to the non-prioritised rank. With the chisq scoring function, the ranked gene list achieved an average enrichment of 3.65 folds (maximum 28.3 folds) for SA-2603 and 3.16 folds for EC-K12 (maximum 27 folds). The probability enrichments of other validation sets are listed in Tables 1 and 2. High AUC values were obtained from the stratified cross-validations of inductive CGP experiments. In particular, SVM achieved near-perfect AUCs in both SA-2603 C and B validation sets, whereas ADTree had the best AUC of 0.975 in M set genes (the trained ADTree model is shown in Additional file 8). Similarly, the best AUC was achieved by SVM/RBF in the EC-K12 M validation set (0.964). The best AUCs of 0.998 and 0.981 were also achieved in the rediscovery of C and B set genes (by IBk and ADTree respectively).

Simulations were performed to investigate the effect of number of genome examples on statistical CGP performance. The range of AUCs was found to be considerably broader with fewer genome examples (Figure 2). However, a median AUC (>0.95) could still be achieved with as few as 10 genome examples in each group in the rediscovery of the M set genes, compared with a maximum of 0.97 using all 417 bacterial genomes (Figure 3), indicating that the method has considerable power using even a very small genome sample.

The same simulation performed on inductive CGP (with SVM/Poly) also achieved an median AUC >0.90 with only 20 genome examples (Figure 4). It was noted, however, that the AUCs peaked at 70 to 160 genome examples (with corresponding AUCs between 0.93–0.95) and the performance gradually declined as more genome examples were added to the profile. Considerable variation of AUC was also noted when the full 417 genomes were included in the profile panel (median AUC: 0.872; interquartile range: 0.858–0.898; range: 0.824–0.925).

Statistical CGP on the anaerobic mixed-acid fermentation rediscovery task for EC-K12 performed poorly (AUC: 0.46–0.77). However, the maximum probability enrichment was high (up to 108-folds, Table 3). Bacterial genes specific to anaerobic metabolism were identified with high ranking scores (the pfl complex: above 0.27 pct; adhE: 2.1 pct; ackA: 2.1 pct; pta: 12 pct; see Additional files 9 and 10) by amss. In contrast, genes shared with aerobic respiration, such as the fumerase genes (fumABC) and the phosphoenolpyruvate carboxylase gene (ppc), were ranked much lower (61–96 pct and 57 pct respectively). For genes encoding the fumarate reductase complex, there were mixed results: the membrane anchor subunits (frdCD) were ranked highly (10.1 and 7.8 pct respectively) and the catalytic subunits were placed at the bottom of the rank (frdAB, 93 and 99 pct). Better overall performance of inductive CGP was achieved compared with statistical CGP (AUC: 0.70–0.86). The best AUCs with inductive prioritisation were produced by IBk and SVM/Poly algorithms respectively (0.86 and 0.85).

Inductive CGP was conducted on 31 KEGG pathways of SA-2603 using 7 algorithms. The best supervised machine learning algorithms identified 14 pathways (45%) with AUCs >0.90 and 28 pathways (87%) with AUCs >0.80 (Figure 5 and see Additional file 11). The best performing algorithm was IBk which had the highest AUC in 10 pathways. ADTree and SVM/Poly also performed well, with each producing the best AUC in 8 pathways. SVM/RBF achieved best AUC in 4 pathways. NB and J48 did not produce a best AUC in any of the 31 pathways studied.

In this paper, we applied two approaches (statistical and inductive CGP) to prioritise candidate genes for functional discovery, based on the occurrence patterns of candidate genes in a selected set of bacterial genomes (phylogenetic profiles). Our findings demonstrate that both CGP methods can rediscover genes with high accuracy in two selected genomes of E. coli K-12 and S. agalactiae 2603 (Figure 3).

Interestingly, these methods seem relatively insensitive to the number of genome examples. In the peptidoglycan example with statistical CGP (case study 1), we were able to identify peptidoglycan genes with high accuracy, despite only a limited number of sequenced genomes among negative examples. For inductive CGP, increasing profile dimension beyond 200 genome examples apparently resulted in a decrease in subsequent median AUCs, implying that a proportion of genome examples was less informative and might not have contributed to the identification of genes of interest. This finding coincides with the observation by Johti et al., where increasing the phylogenetic profile dimension with redundant genomes did not necessarily improve the accuracy in eukaryotic gene function prediction 26.

In statistical CGP, we found that the scoring functions measuring gene occurrence in both positive and negative genome example groups (amss, hmss, chisq, and bchisq) consistently outperform the scoring functions that measure only positive (sens and ppv), negative (spec and npv), or partial (F-measure) frequencies of the groups. This finding highlights the importance of including both positive and negative examples in comparative genomic studies. In inductive CGP, the rediscovery experiments favoured ADTree, IBk, and SVM s when compared with other algorithms. In case study 1, the performance of the best inductive and statistical CGP methods are comparable, suggesting that both approaches are capable of producing robust results.

Our results demonstrated that statistical CGP can discover genes specific to a particular function or pathway. For example, by comparing the cell-walled bacteria with cell wall-less Mycoplasma spp. (case study 1), it was expected that the peptidoglycan genes, which are specific for the phenotypic trait, were among the genes lost in this evolutionary lineage 27 (See Additional file 12). Such genes were ranked very highly and yielded favourable aggregated performance (with AUC >0.95). In contrast, results from our anaerobic fermentation experiment (case study 2) suggested that genes specific to anaerobic respiration were placed very highly on the rank (η_max > 95-fold), whereas genes sharing with the obligatory aerobic bacteria (negative examples) were ranked much lower. As these shared genes are present in both phenotypic groups, finding these "shared" genes by applying only statistical CGP is a challenging task. Alternative methods are needed to aid in the discovery of such non-specific genes.

Both occurrence-based CGP methods performed well, suggesting the genes encoding for a complex phenotype are frequently co-present and co-absent across the genomes, forming specific occurrence patterns, thus allowing the functional predictions. This co-occurrence phenomenon may reflect the process of natural selection and the adaptation of microorgranisms into different evolutionary niches.

For bacteria undergoing positive selection, the acquisition of a particular gene group may result in phenotypes conferring survival advantage for the microorganism to adapt to a new environment. It has been known that genes contributing to symbiosis or pathogenesis are frequently organised into genomic islands, in which gene mobility is facilitated by horizontal gene transfer, conferring the ability to form a new relationship with the host 28. The good AUC achieved by inductive CGP in KEGG pathways (case study 3) suggests specific functional co-occurrence patterns of genes do exist, regardless of the physical proximity of the genes or the presence of a mobile genetic structure.

Similarly, negative selection can also contribute to the co-absence of functional gene units across multiple genomes. For a complex phenotype encoded by multiple genes, the deletion of a critical gene could result in the non-expression of phenotype, leading to the subsequent loss of other non-functional genes over time. Thus, the differential co-occurrence patterns in genes can be exploited for comparative genomics studies, as demonstrated by our methods, in assisting our understanding of gene functions.