Departament de Física i Enginyeria Nuclear, Universitat Politècnica de Catalunya, Edifici GAIA, Rambla de Sant Nebridi s/n 08222, Terrassa, Barcelona, Spain

School of Crystallography, Birkbeck College, University of London, Malet Street, London WC1E 7HX, UK

MRC Biostatistics Unit, Robinson Way, Cambridge CB2 0SR, UK

Abstract

Background

Network motifs are small modules that show interesting functional and dynamic properties, and are believed to be the building blocks of complex cellular processes. However, the mechanistic details of such modules are often unknown: there is uncertainty about the motif architecture as well as the functional form and parameter values when converted to ordinary differential equations (ODEs). This translates into a number of candidate models being compatible with the system under study. A variety of statistical methods exist for ranking models including maximum likelihood-based and Bayesian methods. Our objective is to show how such methods can be applied in a typical systems biology setting.

Results

We focus on four commonly occurring network motif structures and show that it is possible to differentiate between them using simulated data and any of the model comparison methods tested. We expand one of the motifs, the feed forward (FF) motif, for several possible parameterizations and apply model selection on simulated data. We then use experimental data on three biosynthetic pathways in

Conclusions

Maximum likelihood as well as Bayesian model comparison methods are suitable for selecting a plausible motif model among a set of candidate models. Our work shows that it is practical to apply model comparison to test ideas about underlying mechanisms of biological pathways in a formal and quantitative way.

Background

Cellular processes are very complex, but it seems that such processes can often be broken down into a small number of reoccuring patterns of interconnections known as

Existing knowledge can be used in the model identification process. If certain aspects of the networks are already known, the space of possible models is smaller and reverse engineering amounts to

In this paper, we wish to show how formal statistical model comparison methods can be applied to a typical systems biology scenario. We analyze whether the dynamic fingerprint of a number of commonly occurring motifs allows discrimination of the underlying network structure based on simulated data. These are the single input motif (SIM), regulatory chain (RC), feedforward (FF) and feedback (FB) motifs. Any of these motif architectures can be parameterized differently when converted to ordinary differential equations resulting in different dynamics of the system. This has been reported in the case of the FF motif

Before we apply model comparison to specific dynamic models derived from biological network motifs (Section Results), we provide some background on statistical model comparison in the next section (Section Methods). The spirit of this work is to demonstrate in a didactic way how different statistical model comparison tools perform on a class of dynamic models of interest in the systems biology community. We have therefore restricted the type and number of methods and models to those most easily implementable and accessible to the non-expert reader. In addition, we provide an introductory exercise with a simple one-equation model in [Additional file

**Supplementary material for "Statistical model comparison applied to common network motifs"**. The example of estimating parameters for a simple DE model consisting of one equation is used to exemplify and discuss statistical issues of model selection as clearly as possible. Results for extended cooperativity effects in models SIM, RC, FF, and FB are also shown.

Click here for file

Methods

A model _{1}(_{K }(^{T }by a system of differential equations _{k}/_{k}(_{ki } of each _{k} are taken at time points _{i}, ^{2}) affects the observations of the variables. For simplicity we assume here that the error distributions of the variables are independent and constant over time. They are also characterised by some variance ^{2 }which we consider to be part of the parameters.

The probability of the data

where _{N }(^{2}) is a Gaussian with mean ^{2}. Estimates _{ki } are obtained by _{k}(_{k}(

Model comparison methods involve two steps: first the model parameters need to be inferred from the available data, and then the adequacy of each calibrated model needs to be assessed. The classical and Bayesian approaches to each step are described next.

Bayesian inference

In Bayesian statistics our knowledge about model parameters, conditional on the observed data, is summarised by probability distributions. This is allowed because parameters are random variables with a degree of uncertainty. The relationship between the data and the parameters is described by

The posterior distribution of the parameters

The denominator in Equation 2, the model evidence, is constant during the calibration step for one particular model and thus can be ignored. However, it becomes our quantity of interest in model comparison. Computation of the model evidence requires solving the integral _{1}, ..., _{S}, from the posterior distribution

where _{s }| _{s}, or to a multivariate

As we show in the simulations below the reciprocal importance sampler is suitable for the comparatively simple models which we investigate in this study. For more complex models, particularly with many modes, more complex model comparison algorithms might be required (see, for example,

Bayesian model comparison

Given a particular candidate model, _{i}, its posterior probability is given by

where _{i}) is the prior probability of the model, and _{i}) is the model evidence which we estimate here by equation 3.

According to Occam's principle, simpler models are preferred over complex models if they explain the data equally well. If the unknown parameters _{1 }| _{2 }| _{1}, _{n }| _{1}, ..., _{n-1}, _{i }of the data _{1}, ..., _{n}) is predicted using earlier parts _{1}, ..., _{i-1 }to calibrate the model. This makes complex models which overfit less likely. An alternative though related measure that penalises models that overfit is the Deviance Information Criterion (DIC)

for samples _{s }from the posterior (for example, by MCMC simulation) and _{s}_{s}, _{s }_{s }is the vector of posterior parameter means. The lower the DIC the better the model. However, we find that the DIC calculations are often unstable resulting in completely unrealistic values, which might be less relied upon.

Note that the complexity of the model is not easily captured by the number of estimated parameters or degrees of freedom, as in the well known Akaike's information criterion (AIC) or even the BIC and related information criteria. If correlated parameters or informative priors are used, for example, the number of

Once the probability of the model is known, we can select the most probable model from a set of competitive models using the Bayes factor _{i})/_{j}). That is, the Bayes factor measures the extent by which the data increase the odds of _{i }to _{j}. Standard cutoffs for interpreting the significance of BFs, like _{i }over _{j}.

The effective degree of freedom measures how many and by how much parameters are constrained by the data. On one hand, each parameter contributes close to one degree if the width of its posterior is small compared to the width of its prior. On the other hand, a parameter contributes very little to the overall effective degrees of freedom if it is not well constrained by the data and the width of its posterior hardly differs from the width of its prior. Consequently, the Bayes factor or effective degrees of freedom cannot eliminate or penalise spurious parameters which are ill determined by the data. It might be dubious to invoke Occam's principle once more (after it has already been incorporated in the model evidence) to decide between models and only additional data should be used for final clarification. For purely pragmatic reasons of convenience one might still accept the model with formally fewer parameters even though it has the same model evidence and same effective degrees of freedom as a more complex one.

Frequentist inference

Parameter estimation by maximising the likelihood of

An unbiased estimator

Confidence intervals for the _{j}, where each particular standard error (SE_{j}) is obtained from the

Model selection as hypothesis testing

A commonly used method for frequentist model comparison is a likelihood ratio test (LRT), in which two nested models with different number of parameters are compared. According to the null hypothesis, a simple model _{s }(with _{s }parameters) is correct, and thus the additional parameters in the more complex model _{c }with _{c }parameters are unnecessary. A ^{2 }distribution with _{c }- _{s} degrees of freedom of the statistic

Applicability of an LRT is limited due to the requirement that the models are nested, and that the parameters are fully identifiable. Akaike's Information Criterion (AIC) allows ranking models even when they are nonnested. It is defined by

where _{i }is the number of parameters in model _{ML }is the value of

Implementation

MCMC methods are generic approaches to obtain samples from posterior distributions without the need to calculate the model evidence in equation 2 (for details see

Five parallel MCMC chains were run for each model. Each chain consisted of 40,000 iterations (20,000 iterations in the simple one-equation model shown in [Additional file

Results

Here we present the model comparison results for different motif architectures (section Common network motifs), and for different parameterizations of the same FF motif (section Variants of feed forward motif). In each case, the models are introduced first, and the statistical comparisons described secondly in terms of Bayesian model evidence, DIC, effective degrees of freedom, and the maximum likelihood value from the MCMC chains. Statistical comparisons are here supported with the use of simulated data. This step is fundamental since it allows us to evaluate the performance of each approach before applying it to experimental data.

Common network motifs

Models

We analyze the identifiability of the four motif architectures shown in figure

Network motifs

**Network motifs**. a) Single input motif (SIM), b) regulatory chain motif (RC), c) feed forward motif (FF), and d) feedback motif (FB). The abbreviations are:

These motifs can be expressed mathematically in many forms. As a first approximation, they are described here as systems of first order ordinary differential equations (ODEs), like those shown in table _{zy }to zero in model FF renders it the same as model SIM), thus they have been boxed together. The same phenomenon is observed between models RC and FB. The same model architectures described assuming cooperative production functions are given in [Additional file

Motif models.

**motif**

**model**

SIM

_{y}_{y}

_{z}_{z}

RC

_{y}_{y}

_{z}_{z}

FF

_{y}_{y}

_{zs}_{zy}_{z}

FB

_{z}_{z}

Simple ODE models for the network motifs of figure 1.

Analysis

Time series data were simulated from each of the models in tables _{y }= _{y }= 1. Following

The choice of parameter priors is critical in Bayesian model comparison. To make comparison as fair as possible, the same distribution was chosen for all rate constants in the simulations below, namely a log-normal distribution with mean 0 and standard deviation 1 in log-space. Finally, for the prior on the noise variance ^{2 }an inverse Gamma distribution with shape

Table

Model comparison results from simple network motifs. _{}

**data source**

**measure**

**SIM**

**RC**

**FF**

**FB**

SIM

log _{i})

**89.3**

73.74

**89.6**

49.03

DIC

**-198.1**

-192.5

**-198.8**

-177.51

pD

3.93

2.94

4.01

4.34

log _{i})

**102.99**

**102.44**

**103.45**

100.70

AIC

**-197.98**

**-196.88**

**-196.9**

-191.4

RC

log _{i})

29.21

**87.61**

73.58

55.38

DIC

-86.17

**-194.60**

-187.13

-175.21

pD

4.08

3.92

4.53

4.66

log _{i})

47.18

**101.22**

100.46

97.62

AIC

-86.36

**-194.44**

-190.92

-185.24

FF

log _{i})

80.20

57.60

**93.43**

22.95

DIC

-184.7

-153.1

**-208.8**

-131.53

pD

4.06

3.92

4.81

5.01

log _{i})

96.42

81.03

**109.17**

77.64

AIC

-184.84

-154.06

**-208.34**

-145.28

FB

log _{i})

-17.60

-13.93

-39.68

**79.07**

DIC

2351.3

2718.1

2375.8

**-181.37**

pD

4.04

3.66

4.61

4.98

log _{i})

-1171.59

-1355.33

-1176.64

**95.62**

AIC

2351.2

2718.66

2363.26

**-181.24**

Model comparison results for artificial data from the simple ODE models SIM, RC, FF (type 1 coherent with OR gate) and negative FB motifs. Each fit is assessed in terms of model evidence, log p(Y | M_{i}

Bayesian model evidence values clearly favour the true model when data is generated from the RC motif (second row in table

We performed the same analysis assuming the motif models could be defined with cooperative production terms (see [Additional file

Variants of feed forward motif

Models

The generic FF architecture shown in figure

Feed forward motif subtypes: coherent and incoherent

**Feed forward motif subtypes: coherent and incoherent**. In a feed forward (FF) motif, the interaction between the

In _{Sy }and Hill coefficient

where ^{+}(_{Sy}, ^{h}/(^{h}). If S were a repressor, the associated Hill function would be ^{-}(_{Sy}, ^{h}). The rate of change of

where ^{+}(_{Sz}, ^{+}(_{yz}, ^{+}(_{Sz}, ^{-}(_{yz},

FF behaviours have been experimentally explored for the coherent (AND gate), coherent (OR gate) and incoherent (AND gate) subtypes in the arabinose, flagella, and galactose systems of

Arabinose system

The arabinose (ara) system is the set of genes that allows intake of the sugar arabinose (figure

Bacterial feed forward systems

**Bacterial feed forward systems**. The bacterial feed forward systems analysed here are the arabinose system (a), the flagella network (b), and the galactose system (c). Figures adapted from

Here we test whether the generic FF.C1.AND model described in

where AraC is abbreviated as AC, and the product of gene _{ac }and _{ab}, are small compared to the levels reached upon activation

Control model for arabinose system

In

This control model is set up as follows: an ON step consists of CRP = 1, lactose = 1, LZ(0) = 0, and an OFF step consists of CRP = 0, lactose = 1, LZ(0) = 1. As before, _{lz }was assumed to be 0. In terms of the target component

Flagella system

The flagella biosynthesis network of

An initial candidate model for the flagella network, model FF.C1.OR.1, is based on the FF.C1.OR equations described in

where FlhDC is abbreviated as FD, FliA as FA, and the product of gene

Control model for flagella system

In

The initialization conditions are FD = 1, FL(0) = 0 for the ON step, and FD = 0 and FL(0) = 1 for the OFF step.

Additional flagella models

We consider two alternative models for the flagella FF motif incorporating time delays. The interval needed for FliA activation is explicitly modeled via a time delay,

In this type of models, delay differential equation models, the concentration of FA used in the equation at time

Finally, a last model for the flagella system was tested in which the time delay affects both the expressions for FA and FL, model FF.C1.OR.3:

Galactose system

In an incoherent FF motif, the two regulation paths flowing from the master regulator display opposite signs. In particular, in a type 1 incoherent FF motif, the branches from

The equations suggested here to model the gal system are

where GalS is abbreviated as GS, and the product of gene

Control model for galactose system

The same control module developed for the arabinose system was used as a control for the gal system in

where LI refers to the repressor LacI. The simulation conditions are CRP = 1, LI = 1, LZ(0) = 0.

Analysis

Here we first explore the different FF motif subtypes using simulated data generated from equations 15, 18 and 24 under the same conditions used in the experiments: the same signal function, number of observed variables and dataset size. The latter implies that the number of data points for each of the FF systems is: 60 for the arabinose system (30 in ON, 30 in OFF step), 37 for the flagella system (20 in ON, 17 in OFF step), and 15 for the galactose system (only ON step). Note that only one variable -

As shown in table

MCMC model comparison results for the artificial FF datasets.

**data source**

**measure**

**CONTROL**

**FF.C1.AND**

**(Eqns 15)**

**FF.C1.OR.1**

**(Eqns 18)**

**FF.I1.AND**

**(Eqns 24)**

(Eqn. 17)

FF.C1.AND

(Eqns 15)

log _{i})

85.98

**108.94**

106.51

86.53

DIC

-217.98

**-388.91**

-292.81

-205.64

log _{ML}, _{i})

111.22

**165.18**

164.34

111.15

AIC

-208.44

**-316.36**

-314.68

-208.3

FF.C1.OR.1

(Eqns 18)

log _{i})

(Eqn. 20)

40.12

30.90

**48.09**

42.55

DIC

-108.96

-102.47

**-136.65**

-117.22

log _{ML}, _{i})

62.73

60.06

**86.94**

69.97

AIC

-111.46

-106.12

**-159.88**

-125.94

FF.I1.AND

(Eqns 24)

log _{i})

(Eqn. 26)

12.52

8.28

**13.78**

**14.02**

DIC

-38.33

-35.75

-36.75

**-41.63**

log _{ML}, _{i})

26.66

25.69

**30.09**

**30.86**

AIC

-39.32

-37.38

**-46.18**

**-47.72**

Datasets have the same number of samples as the experimental data from

MCMC model comparison results for the experimental FF datasets.

**data source**

**measure**

**CONTROL**

**FF.C1.AND**

**FF.C1.OR.1**

**FF.I1.AND**

ara system

log _{i})

51.22

**74.99**

72.31

51.88

DIC

-158.69

-435.02

**-445.81**

-275.31

log _{ML}, _{i})

83.99

140.05

**147.06**

83.99

AIC

-157.98

-264.10

**-278.12**

-151.98

flagella system

log _{i})

**-16.22**

-264.64

-73.81

DIC

-54.99

**-41275.70**

-2296.40

5878.25

log _{ML}, _{i})

**31.02**

-15.28

-4.72

-256.50

AIC

**-52.04**

46.56

25.44

529.20

gal system

log _{i})

-8.71

-13.29

-4.49

**-0.42**

DIC

10.53

9.88

-3.53

**-20.47**

log _{ML}, M_{i})

-1.79

-1.13

11.57

**12.53**

AIC

13.58

18.26

-7.14

**-9.04**

The equations used are the same as in table 3.

Table

Reconstructing the arabinose dataset

**Reconstructing the arabinose dataset**. Model predictions for the arabinose dataset, using the posterior parameter values inferred with MCMC. The fractional value of the feedforward element

In the case of data from the flagella network, the results unexpectedly favour the control model. Since this network is indeed believed to be composed of a feed forward FF.C1.OR motif

Assessment of additional flagella models for the flagella dataset

**Measure**

**CONTROL**

**FF.C1.OR.1**

**FF.C1.OR.2**

**FF.C1.OR.3**

log _{i})

**-16.22**

-73.81

-179.11

-33.81

DIC

-54.99

**-2296.40**

23.59

-212.89

log _{ML}, _{i})

31.02

-4.72

30.38

**35.09**

AIC

-52.04

23.44

-46.76

**-54.18**

The prior for the delay parameter was a Normal distribution with mean and variance 10, truncated at [0,100].

Finally, in the case of data from the

Discussion

When building a model for a biological system, one has to decide whether to use parameter values from the literature or estimate them from the data. While the first option may be very useful, one has to bear in mind the limitations that extrapolating parameter values from other systems and experimental conditions have

Here we have provided a short overview of the Bayesian and frequentist approaches to model comparison. Then, we have applied the model comparison techniques to two cases. First, we have investigated the identifiability of a series of transcription regulation motif architectures (SIM, RC, FF and FB). The objective was to find out if one could infer the correct underlying model structure given time series data from each of these motifs (tables

Secondly, it is known that the same model architecture can give rise to different dynamics depending on the particular model parameterization. To explore this issue, we have focused on the FF motif, an architecture for which extensive experimental caracterization has been carried out during the past years

The motivation behind the original papers

While the

It could be argued that the body of information assumed available to generate the dataset used is so large that no model uncertainties remain. We wish to stress that embarking on a model comparison exercise is a way to make sure that all relevant mechanisms have been accounted for. Therefore, model comparison strategies should be regarded as

Conclusions

We have given an overview of model comparison methods suitable for selecting a plausible network motif structure among a set of candidate models for time series data on gene regulation. We show that it is practical to apply maximum likelihood as well as Bayesian model comparison procedures to test ideas about underlying mechanisms of biological pathways in a formal and quantitative way.

Authors' contributions

All authors participated in the design of the study, helped to draft the manuscript. NDP performed the statistical analysis and mainly drafted the manuscript. LW coordinated the project. All authors read and approved the final manuscript.

Acknowledgements

This work was mainly supported by The Wellcome Trust's Functional Genomics Integrated Thematic Programme, and primarily carried out in the School of Crystallography at Birkbeck College (London, UK). The authors wish to acknowledge the Biostatistics Unit at the MRC (Cambridge, UK) for covering the publication costs. NDP acknowledges current funding by the Instituto de Salud Carlos III (REEM project RD07_0060_0017), the Ministerio de Ciencia e Innovación of Spain (FIS2009-13360), and the Generalitat de Catalunya (AGAUR 2009SGR1168).