Bayesian networks 4 5 6 are one of the most popular modeling approaches for network construction using gene expression data 7 8 9 10 11 12 13 14 15 16 17 . A Bayesian network is a probabilistic graphical model for which the joint distribution of all the nodes is factorized into independent conditional distributions of each node given its parents. The goal of Bayesian network inference is to arrive at a directed graph such that the joint probability distribution is optimized globally. While different Bayesian network structures may give rise to the same probability distribution, so that such networks in general do not imply causal relationships, prior information can be used to break this nonidentifiability so that causal inferences can be made. For example, systematic sources of perturbation such as naturally occurring genetic variation in a population or specific drug perturbations in which response is observed over time can lead to reliable causal inference 1 2 18 19 . A Bayesian network is a directed acyclic graph (DAG). Therefore, cyclic components or feedback loops cannot be accommodated. This DAG constraint is an obstacle to using the Bayesian network approach for modeling gene regulatory networks because feedback loops are typical in many biological systems 20 . The DAG constraint is removed when dynamic Bayesian networks are used to model time-series expression data 19 21 22 23 24 . Dynamic Bayesian networks represent genes at successive time points as separate nodes, thus allowing for the existence of cycles. Bayesian network construction is an NP-hard problem 25 26 , with computational complexity increasing exponentially with the number of nodes considered in the network construction process. In spite of some attempts to reduce the computational cost 27 , the Bayesian network approach in general is computationally intensive to implement, especially for network inference on a genome-wide scale.

In regression-based methods, network construction is recast as a series of variable selection problems to infer regulators for each gene. The greatest challenge is the fact that there are usually far more candidate regulators than observations for each gene. Some authors have used singular value decompositions to regularize the regression models 28 29 30 . Others have built a regression tree for each target gene, using a compact set of regulators at each node 31 32 33 34 . Huang et al. 35 used regression with forward selection after pre-filtering of candidates deemed irrelevant to the target gene, and Imoto et al. 16 used nonparametric regression embedded within a Bayesian network. L1-norm regularization, including the elastic net 36 37 and weighted LASSO 38 , has also been widely used 39 40 41 42 43 44 45 46 47 48 49 .

Ordinary differential equations (ODE) provide another class of network construction strategies 50 51 52 53 . Using first-order ODEs, the rate of change in transcription for a target gene is described as a function of the expression of its regulators and the effects caused by applied perturbations. ODE-based methods can be broadly classified into two categories, depending on whether the gene expressions are measured at steady state 54 55 56 57 58 or over time 51 52 53 . As an example, the TSNI (Time Series Network Identification) algorithm used ODEs to model time series expression data subject to an external perturbation 53 . To handle the dimensionality challenge (i.e. the number of observations per gene is much smaller than the number of genes), Bansal et al. employed a cubic smoothing spline to interpolate additional data points, and applied Principal Component Analysis to reduce dimensionality.

To help mitigate problems with using gene expression data in network inference, external data sources can be integrated into the inference process. Public data repositories provide a rich resource of biological knowledge relevant to transcriptional regulation. Integrating such external data sources into network inference has become an important problem in systems biology. James et al. 43 incorporated documented experimental evidence about the presence of a binding site for each known transcription factor (TF) in the promoter region of its target gene in Escherichia coli. Djebbari and Quackenbush 13 used preliminary networks derived from literature indexed in PubMed and protein-protein interaction (PPI) databases as seeds for their Bayesian network analysis. Zhu et al. 59 showed that combining information from TF binding sites and PPI data increased overall predictive power. Geier et al. 15 examined the impact of external knowledge with different levels of accuracy on network inference, albeit on a simulated setting. Imoto et al. 16 described different ways to specify knowledge about PPI, documented regulatory relationships and well-studied pathways as prior information. Lee et al. 44 presented a systematic way to include various types of biological knowledge, including the gene ontology (GO) database, ChIP-chip binding experiments and a compressive collection of information about sequence polymorphisms.

This article is an extension of Yeung et al. 3 which adopted a regression-based framework in which candidate regulators are inferred for each gene using expression data at the previous time point. Iterative Bayesian model averaging (iBMA) 60 61 62 was used to account for model uncertainty in the regression models. A supervised framework was used to estimate the relative contribution of each type of external knowledge and from this a shortlist of promising regulators for each gene was predicted. This shortlist was used to infer regulators for each gene in the regression framework.

Our contributions are four-fold. First, we develop a new method called iBMA-prior that explicitly incorporates external biological knowledge into iBMA in the form of a prior distribution. Intuitively, we consider models consisting of candidate regulators supported by considerable external evidence to be frontrunners. A model that contains many candidate regulators with little support from external knowledge is penalized. Second, we demonstrate the merits of specifying the expected number of regulators per gene as priors through iBMA-size, which is a simplified version of iBMA-prior without using gene-specific external knowledge. Third, we refine the supervised framework to adjust for sampling bias towards positive cases in the training data, thereby calibrating the prior distribution. Fourth, we expand our benchmark to include simulated data, and compare our iBMA methods to L1-regularized regression-based methods. Specifically, we applied iBMA-prior to real and simulated time-series gene expression data, and found that it out-performed our previous work 3 and other leading methods in the literature on these data, producing more compact and accurate networks. Figure 1 summarizes iBMA-prior and our main contributions.

First, we assessed the improvement of iBMA-prior over that of our previous work iBMA-shortlist from Yeung et al. 3 (see Methods for details) when applied to the same yeast time-series gene expression data. Then, we compared our BMA-based methods to several L1-regularized methods, including the least absolute shrinkage and selection operator (LASSO) 36 63 and least angle regression (LAR) 64 . Regularized regression methods combine shrinkage and variable selection. L1-regularized methods aim to minimize the sum of squared errors with a bound on the sum of the absolute values of the coefficients 65 . Efficient implementations are available for some of these methods, including LASSO and LAR, and these methods have been applied to high-dimensional data in which there are more variables than observations 64 66 67 .

We also compared the performance of our method with and without using external biological knowledge. We assessed hybrid methods by combining LASSO and LAR with the same supervised learning stage that was used in iBMA-prior and iBMA-shortlist. Table 1 lists all the methods compared in this analysis.

To evaluate the accuracy of the network constructed by each method, we assessed its concordance with the Yeastract database, a curated repository of regulatory relationships between known TFs and target genes in the Saccharomyces cerevisiae literature 68 . If a regulatory relationship documented in Yeastract was also inferred in the network, we concluded that this relationship was recovered by direct evidence. Some of the positive examples used in the supervised learning stage are also documented in Yeastract. To avoid bias, we did not consider those regulatory relationships in the assessment. For each method compared, we applied Pearson’s chi-square test to a 2 × 2 contingency table that quantified the concordance of the inferred network with the Yeastract database. We also computed the true positive rate (TPR), defined as the proportion of the inferred positive relationships that are documented in Yeastract. It should be noted that Yeastract cannot document all “true” relationships as the entire set of regulatory relationships in yeast has yet to be defined. We further considered the ratio of the observed number of recovered relationships to its expected count as a result of random assortment (O/E). More detailed definitions of the assessment criteria can be found in Additional file 1: Figure S1.

Table 2 summarizes the assessment results for the nine methods compared. Additional details are presented in Additional file 2: Table S1. First, we studied the impact of integrating external knowledge into the network construction process under the iBMA framework. The TPR of iBMA-prior was 18.00%, and the number of recovered positive relationships was 593, which is 4.11 times more than the expected number by random chance. Using the revised supervised step described in this work without incorporating prior probabilities into the iBMA framework, iBMA-shortlist yielded a TPR of 12.78% and O/E ratio of 2.92. This is an improvement over network A (TPR = 9.98% and O/E = 2.28) constructed using the same algorithm and our previous version of the supervised framework as described in Yeung et al. 3 . All of our methods that incorporate external knowledge (iBMA-prior, iBMA-shortlist and network A) produced higher TPRs than iBMA-noprior for which only the time-series gene expression data were used. In particular, iBMA-prior produced a TPR (18.00%), which represents a two-fold increase over iBMA-noprior (8.9%). Therefore, the integration of external data clearly improved the recovery of known relationships, and our latest method, iBMA-prior, performed the best.

^a The p-value of Pearson’s chi-square test measures the strength of association between an inferred network and the Yeastract database.

^b True positive rate (TPR) is defined as the proportion of inferred regulatory relationships that are documented in Yeastract.

^c The number of misclassified cases is the sum of false positives and false negatives.

^d The O/E ratio is the number of folds the observed number of recovered relationships (i.e., TP) in excess of the expected count of recovery by chance.

Next, we compared our iBMA-based methods to L1-regularized methods. All the approaches that used LASSO and LAR generated networks that had far more mis-classifications than the iBMA-based methods. Specifically, applications of LASSO or LAR without the supervised framework (LASSO-noprior and LAR-noprior) had TPRs of 5.20% and 7.71% respectively, the lowest among all the methods considered. Incorporating external knowledge did improve both LASSO and LAR, increasing the TPRs to about 11% in both LASSO-shortlist and LAR-shortlist. However, these TPRs were still lower than the TPRs for our iBMA-based methods. Our iBMA-based methods therefore outperformed methods based on LASSO and LAR for these data.

Finally, we investigated the impact of priors in iBMA-size, in which we applied a model size prior to calibrate the sparsity of the inferred networks without using any external data sources. iBMA-size can be considered as a simplified version of iBMA-prior that sets the regulatory potential (the prior probability that a candidate regulates a given gene) to a constant parameter that controls the expected number of regulators per gene. From Table 2, iBMA-size produced a TPR of 16.84%, which was higher than all the other methods considered except iBMA-prior. Although the number of recovered positive relationships was lower than that of iBMA-prior (114 <593), iBMA-size also produced a network that was more compact (17,202 edges compared to 21,951 edges). We would recommend iBMA-size when gene-specific external information is not available.

In Table 2 and Additional file 2: Table S1, all the iBMA networks were thresholded at a posterior probability of 50% (i.e., edges with posterior probability <50% were removed). We found that iBMA-prior also out-performed other methods for these data over different posterior probability thresholds (see Additional file 2: Table S2).

In another assessment, we checked whether the set of target genes containing known binding sites for a certain TF were enriched among the child nodes of that TF in each inferred network. We first extracted the known binding sites for 129 TFs documented in the JASPAR database 69 70 . Using TFMscan 71 , we retrieved a set of genes containing the known binding sites in their upstream regions for each TF. We then checked for enrichment of these genes among the inferred child nodes of the corresponding TFs in each network with Fisher’s exact test. Table 3 reports the number of TFs whose inferred child nodes exhibited such enrichment, at a false discovery rate (FDR) of 10%. All of the methods that made use of external information outperformed all of those that did not, illustrating the benefit of incorporating external knowledge. LASSO-shortlist and LAR-shortlist appeared to produce slightly better results than iBMA-prior in this binding site analysis, but it is likely the consequence of their larger network sizes (>2x larger than iBMA prior).

^a FDR was controlled at 10%.

Lee et al. 44 proposed a regression-based network construction method called Lirnet, which performed well on a publicly available gene expression data set from Brem et al. 72 . The Brem data set recorded the steady-state expression levels for 112 yeast segregants, 95 of which were profiled in our time-series experiments under different growth conditions. Lee et al. 44 showed that Lirnet out-performed Bayesian networks on the same data, and so we compared our top performer, iBMA-prior, with Lirnet. Because Lirnet was formulated to analyze steady-state expression data with no time components, we adapted our method to static data by removing the subscript referring to the time point from Equation (4):

E [ X g , s | D ] = β g , 0 + ∑ r ∈ R g β g , r X r , s ,

We applied iBMA-prior to the same 3152-gene subset of the Brem et al. data that Lee et al. 44 used. Lirnet constrained the search of regulators for each target gene to 304 known TFs. For fair comparison, we also confined the set of candidate regulators to the same TFs. Networks constructed from steady-state gene expression data cannot have feedback loops 73 74 75 . To detect and remove such loops from our inferred network, we identified all strongly connected components using the igraph R package, and deleted the TF-gene link associated with the lowest posterior probability for each cycle.

Same as before, we evaluated different methods by assessing the concordance of the inferred networks with the Yeastract database using Pearson’s chi-square test. The assessment results in Table 4 show that iBMA-prior outperformed Lirnet, almost doubling the TPR and the O/E ratio while producing a comparable number of misclassified regulatory relationships.

^a The p-value of Pearson’s chi-square test measures the strength of association between an inferred network and the Yeastract database.

^b True positive rate (TPR) is defined as the proportion of inferred regulatory relationships that are documented in Yeastract.

^c The number of misclassified cases is the sum of false positives and false negatives.

^d The O/E ratio is the number of folds the observed number of recovered relationships (i.e., TP) in excess of the expected count of recovery by chance.

We designed and conducted a series of simulations to further assess our proposed method. We used the fitted model obtained from applying iBMA-prior to the yeast time-series microarray data set as the true underlying network, and generated simulated expression data from the estimated linear regression model. Twenty data sets, each with the same dimensions as the real time-series expression data, were independently generated as follows:

1. Set the prior probability of a regulatory relationship for each gene pair to the same value as the regulatory potential obtained at the supervised learning stage using the real external data.

2. Set the expression levels of the 3556 genes for the 95 yeast segregants and the two parental strains at time t = 0 as the observed measurements in the real yeast time-series gene expression data.

3. For each target gene g, define the set R _g of true regulators as those with a posterior probability of ≥50% in our inferred network using iBMA-prior and the real time-series data.

4. For time t = 1 to 5,

For gene g = 1 to 3556, generate the simulated true expression level for each segregant s using the following equation:

X g , t , s true = β g , 0 + ∑ r ∈ R g β g , r X r , t − 1 , s true ,

where the β’s are given by the posterior expectation of the regression coefficients corresponding to the set of true regulators determined in Step 3.

5. Generate the simulated observed gene expression levels by adding noise to the true expression levels without measurement errors, i.e.,

X g , t , s = X g , t , s true + ϵ g , t , s ,

where ϵ _g,t,s  ~ N(0, σ _g ²) with σ _g ² being given by the sample variance of the regression residuals in the real data analysis. Others, e.g. 76 , have shown that the error in log ratios of expression data is reasonably approximately by a normal distribution.

To assess the accuracy of networks inferred with the simulated data sets, we compared each of these networks to the true network created in Step 3 of the data generation algorithm. We used the same assessment criteria as in the real data analysis with the true network replacing Yeastract as the reference. As shown in Table 5, iBMA-prior out-performed the other iBMA-based methods, yielding a TPR of 71.13% averaged over 20 replications (compared to 47.23% for iBMA-shortlist, 20.31% for iBMA-size, and 8.55% for iBMA-noprior).

^a The p-value of Pearson’s chi-square test measures the strength of association between an inferred network and the true network for the simulation study.

^b True positive rate (TPR) is defined as the proportion of correctly inferred regulatory relationships.

^c The number of misclassified cases is the sum of false positives and false negatives.

Remark: The values reported in the table were averaged across the 20 replications. The true network for the simulation study contained a total of 21951 edges.

We applied our method to a set of time-series mRNA expression data measuring the gene expression levels of 95 genotyped haploid yeast segregants perturbed with the macrolide drug rapamycin 3 . These segregants, along with their genetically diverse parents, BY4716 (BY) and RM11-1a (RM), have been genotyped previously 72 . Rapamycin was chosen for perturbation because it was expected to induce widespread changes in global transcription, based on a screen of the public microarray data repositories 78 79 80 . This perturbation allowed for the capture of a large subset of all regulatory interactions encoded by the yeast genome. Each yeast culture was sampled at 10-minute intervals for 50 minutes after rapamycin addition. The RNA purified from these samples was profiled with Affymetrix Yeast 2.0 microarrays. Probe signals were summarized into gene expression levels using the Robust Multi-array Average (RMA) method 81 and genes not exhibiting significant changes in expression were filtered from the data as described in 3 . The data subset that remained consisted of the time-dependent mRNA expression profiles of 3556 genes. The complete time series gene expression data are publicly available at ArrayExpress (http://www.ebi.ac.uk/arrayexpress/) with accession number E-MTAB-412.

BMA is a variable selection approach that takes model uncertainty into account by averaging over the posterior distribution of a quantity of interest based on multiple models, weighted by their posterior model probabilities 82 83 . In BMA, the posterior distribution of a quantity of interest Θ given the data D is given by Pr Θ | D = ∑ k = 1 K Pr Θ | D , M k Pr M k | D , where M ₁,…,M _k are the models considered. Each model consists of a set of candidate regulators. In order to efficiently identify a compact set of promising models M _k out of all possible models, two approaches are sequentially applied. First, the leaps and bounds algorithm 84 is applied to identify the best nbest models for each number of variables (i.e., regulators). Next, Occam’s window is applied to discard models with much lower posterior model probabilities than the best one 85 . The Bayesian Information Criterion (BIC) 86 is used to approximate each model's integrated likelihood, from which its posterior model probability can be determined.

While BMA has performed well in many applications 60 , it is hard to apply directly to the current data set in which there are many more variables than samples. Yeung et al. 62 proposed an iterative version of BMA (iBMA) to resolve this problem. At each iteration, BMA is applied to a small number, say, w = 30, of variables that could be efficiently enumerated by leaps and bounds. Candidate predictor variables with a low posterior inclusion probability are discarded, leaving room for other variables in the candidate list to be considered in subsequent iterations. This procedure continues until all the variables have been processed.

We formulated network construction from time series data as a regression problem in which the expression of each gene is predicted by a linear combination of the expression of candidate regulators at the previous time point. Let D be the entire data set and X _g,t,s be the expression of gene g at time t in segregant s. Denote by R _g the set of regulators for gene g in a candidate model. The expression of gene g is formulated by the following regression model:

E [ X g , t , s | D ] = β g , 0 + ∑ r ∈ R g β g , r X r , t − 1 , s ,

where E denotes expectation and β’s are regression coefficients. For each gene, we apply iBMA to infer the set of regulators.

To account for external knowledge in the network construction process, Yeung et al. 3 introduced a supervised framework to estimate the weights of various types of evidence of transcriptional regulation and subsequently derived top candidate regulators. For instance, a target gene is likely to be co-expressed with its regulators across diverse conditions in publicly available, large-scale microarray experiments 78 87 88 . ChIP-chip data 89 provide supporting evidence for a direct regulatory relationship between a given TF and a gene of interest by showing that the TF directly binds to the promoter of that gene. A candidate regulator with known regulatory roles in curated databases such as the Saccharomyces Genome Database (SGD) 90 would be favored a priori. Polymorphisms in the amino acid sequence of a candidate regulator that affect its regulatory potential provide further evidence of a regulatory relationship 44 . Common gene ontology (GO) 91 annotations for a target gene and candidate regulators also provide evidence of functional relationship.

To study the relative importance of the various types of external knowledge from the supervised framework, we collected 583 positive examples of known regulatory relationships between TFs and target genes from the Saccharomyces cerevisiae Promoter Database (SCPD) 92 and the Yeast Protein Database (YPD) 93 . Random sampling of these TF-gene pairs was used to generate 444 negative examples. Logistic regression using BMA was applied to estimate the contribution of each type of external knowledge in the prediction of regulatory relationships. The fitted model was then used to predict the regulatory potential π _gr of a candidate regulator r for a gene g, i.e., the prior probability that candidate r regulates gene g, for all possible regulator-gene pairs. Next, the regulatory potentials were used to rank and shortlist the top p candidate regulators for each gene (p = 100 by default in our experiments). The shortlisted candidates were then input to BMA for variable selection in the network construction process.

The potential benefit of using information from external knowledge to refine the search for regulators was shown by Yeung et al. and many others 3 13 15 16 17 43 44 . However, external knowledge was only used to shortlist the top p candidate regulators for each target gene in Yeung et al. Here, we develop a formal framework that fully incorporates external knowledge into the BMA network construction process.

We associate each candidate model M _k with a prior probability, namely:

Pr M k ∝ ∏ r π gr δ kr 1 − π gr 1 − δ kr ,

where π _gr is the regulatory potential of a candidate regulator r for a gene g, δ _kr  = 1 if r ∈M _k and δ _kr  = 0 otherwise 85 94 . Intuitively, we consider models consisting of candidate regulators supported by considerable external evidence to be frontrunners. A model that contains many candidate regulators with little support from external knowledge is penalized.

The posterior model probability of model M _k is given by

Pr M k | D ∝ f D | M k Pr M k ,

where f(D | M _k ) is the integrated likelihood of the data D under model M _k , and the proportionality constant ensures that the posterior model probabilities sum up to 1.

Then Occam’s window was used to discard any model M _k having a posterior odds less than 1/OR relative to the model with the highest posterior probability, M _opt . The parameter OR controls the compactness of the set of selected models, and here we set it to 20.

In Yeung et al. 3 , the models selected in an intermediate iteration by iBMA were not recorded once that iteration was completed, and the final set of models selected were chosen only from those considered in the last iteration. While computationally efficient, this strategy overlooked the possibility of accumulated model support over multiple iterations. We improve the model selection process by storing all the models selected in any iteration and applying Occam’s window to this cumulative set of models as the last step in the algorithm.

At the end of each iteration of iBMA, and after applying Occam’s window to all models considered, we compute the posterior inclusion probabilities for each candidate regulator r by summing up the posterior probabilities of all models that involve this regulator.

Pr ( β gr ≠ 0 | D ) = ∑ k : M k ∈ Φ Pr M k | D · δ kr .

where F is the set of all possible models for gene g, β _gr is the regression coefficient of a candidate regulator r for a gene g, δ _kr  = 1 if r ∈M _k and δ _kr  = 0 otherwise. Finally, we infer regulators for each target gene g by thresholding on the posterior inclusion probability at a predetermined level (50% in all our experiments unless otherwise specified).

We have extended the supervised framework of Yeung et al. 3 in three ways.

1. For each gene g, rank the candidate regulators based on the regulatory potentials predicted from the supervised framework.

2. Shortlist the top p candidates from the ranked list (p = 100 in our experiments).

3. Fill the BMA window with the top w candidates in the shortlist (w = 30 in our experiments).

4. Apply BMA with prior model probabilities based on the external knowledge:

a. Determine the best nbest models for each number of variables using the leaps and bounds algorithm (nbest = 10 in our experiments).

b. For each selected model, compute its prior probability relative to the w candidates in the current BMA window using Equation (5).

c. Remove the w candidate regulators with posterior inclusion probability Pr(β _gr  ≠ 0 | D) <5%.

5. Fill the w-candidate BMA window with those not considered yet in the shortlist.

6. Repeat steps 4–5 until all the p candidates in the shortlist have been processed.

7. Compute the prior probability for all selected models relative to all the p shortlisted candidates using Equation (5).

8. Take the collection of all models selected at any iteration of BMA, and apply Occam’s window, reducing the set of models.

9. Compute the posterior inclusion probability for each candidate regulator using the set of selected models, and infer candidates associated with a posterior probability exceeding a pre-specified threshold (50%) to be regulators for target gene g.

External knowledge is used in the following ways:

1. All the candidate regulators are ranked according to their regulatory potentials, which were predicted using the available external data sources at the supervised learning stage.

2. Model selection is performed by comparing models against each other based on their posterior odds. As shown by Equation (6), the posterior odds is proportional to a product of the integrated likelihood and the prior odds. The prior probability and, therefore, the prior odds, of a candidate model are formulated as a function of regulatory potentials.

3. The posterior inclusion probability of each candidate regulator, from which inference is made about the presence or absence of a regulatory relationship, is positively related to its regulatory potential. As shown in Equation (5), a factor of π _gr is contributed to each model in which the candidate g is included. Otherwise, a factor of 1- π _gr is contributed to each model.