Physics Department, University of Freiburg, Hermann Herder Straße 3, 79104 Freiburg, Germany

Freiburg Centre for Biosystems Analysis (ZBSA), University of Freiburg, Habsburgerstraße 49, 79104 Freiburg, Germany

Freiburg Institute for Advanced Studies (FRIAS), University of Freiburg, Albertstraße 19, 79104 Freiburg, Germany

Freiburg Initiative in Systems Biology (FRISYS), University of Freiburg, Schaenzlestraße 1, 79104 Freiburg, Germany

BIOSS Centre for Biological Signalling Studies, University of Freiburg, Schaenzlestraße 18, 79104 Freiburg

Department of Clinical and Experimental Medicine, Universitetssjukhuset, 58183 Linköping, Sweden

Institute of Bioinformatics and Systems Biology, Helmholtz Zentrum München, Ingolstädter Landstraße 1, 85764 Neuherberg, Germany

Abstract

Background

Predicting a system’s behavior based on a mathematical model is a primary task in Systems Biology. If the model parameters are estimated from experimental data, the parameter uncertainty has to be translated into confidence intervals for model predictions. For dynamic models of biochemical networks, the nonlinearity in combination with the large number of parameters hampers the calculation of prediction confidence intervals and renders classical approaches as hardly feasible.

Results

In this article reliable confidence intervals are calculated based on the

Conclusions

The presented methodology allows the propagation of uncertainty from experimental to model predictions. Although presented in the context of ordinary differential equations, the concept is general and also applicable to other types of models. Matlab code which can be used as a template to implement the method is provided at

Background

A major goal of Systems Biology is the prediction of cellular behavior over a broad range of environmental conditions. To be able to generate realistic predictions, the individual processes of a system of interest have to be translated into a mathematical framework. The task of establishing a realistic mathematical model which is able to reliably predict a systems behavior is to comprehensively use the existing knowledge, e.g. in terms of experimental data, to adjust the models’ structures and parameters.

The major steps of this mathematical modeling process comprise model discrimination, i.e. identification of an appropriate model structure, model calibration, i.e. estimation of unknown model parameters, as well as prediction and model validation. For all these topics it is essential to have appropriate methods assessing the certainty or ambiguity of any result for given experimental information.

For parameter estimation, there are several approaches to derive confidence intervals, like standard errors which are based on an estimate of the covariance matrix of the parameter estimates

The mere estimation of parameters is often not the final aim of an investigation. More frequently, it is desired to utilize the parametrized model to generate model predictions such as the dynamic behavior of unobserved components. Classically, the uncertainty in the model parameters is attempted to be translated into corresponding prediction confidence intervals, also called

Methods for an approximate sampling of the parameter space, e.g. the Markov Chain Monte Carlo (MCMC) methods

The idea of the

The theoretical background of the prediction profile likelihood, also called

In this paper, this concept is applied to ODE models occurring in dynamic models, e.g. in Molecular and Systems Biology as well as chemical engineering. In this context the approach a data-based observability analysis is introduced. Moreover, the prediction profile likelihood concept is extended to obtain confidence intervals for validation experiments.

Methods

The methodology presented in the following is general, i.e. not only applicable for ODEs. Therefore, we first introduce the prediction profile likelihood as well as prediction confidence intervals and next illustrating the applicability for ODE models.

The prediction profile likelihood

For additive Gaussian noise ^{2}) with known variance ^{2}, two times the negative log-likelihood

of the data _{
i
}(_{
i
}−_{
i
},^{2}/^{2}. In this case, maximum likelihood estimation is equivalent to standard least-squares estimation

i.e. to minimizing the residual sum of squares.

with an externally controlled input function

The parameter vector

It has been shown

for a parameter _{
j
}given a data set

for the estimation of a single parameter. Here, ^{∗} is the maximum of the log-likelihood function after all parameters are optimized. In (5), the optimization is performed for all parameters except _{
j
}. The analogy of likelihood-based parameter and prediction confidence intervals is discussed in the Additional file

**[Kreutz12_SupplementalMaterial].** In the Supplemental Material, theoretical issues like re-parametrization of the model, coverage, or the accuracy of the asymptotically derived threshold are discussed in detail. Moreover, the computational implementation is described and additional analyses of the two models are provided.

Click here for file

The desired coverage

i.e. the probability that the true parameter value is inside the confidence interval, holds for 6 if the magnitude of the decrease of the residual sum of squares by fitting of _{
j
} is

The _{pred},_{pred }= {_{pred},_{pred},_{pred}} specifying a prediction observable _{pred }evaluated at time point _{pred} given the externally controlled stimulation _{pred}. In principle, every quantity which can be computed based on the model can serve as a model prediction

In some cases the observable _{pred} corresponds to measuring a dynamic variable _{pred}(_{pred }does neither have to coincide with a dynamic variable nor with an observational function

In analogy to (7), the desired property of a prediction confidence interval PCI_{
α
}(

that the true value of _{pred},_{true}) is inside the prediction confidence interval PCI_{
α
}is equal to

The prediction profile likelihood

is obtained by maximization over the model parameters satisfying the constraint that the model response ^{∗}) after fitting is equals to the considered value

i.e. the set of predictions _{pred}

Instead of sampling a high-dimensional parameter space, the prediction profile likelihood calculation comprises sampling of a one-dimensional prediction space by evaluating several predictions

The validation profile likelihood

Likelihood-based confidence interval like (6) or (10) correspond to the set of predictions which are not rejected by a likelihood ratio test. Having a prediction confidence interval, the question arises whether a model has to be rejected if a validation measurement is outside the predicted interval. This, in fact, would hold if a “perfect” validation measurement would be available, i.e. a data point without measurement noise. For validation experiments, however, the outcome is always noisy and is therefore expected to be more frequently outside the PCI than the true value. Hence, the prediction confidence interval (10) has to be generalized for application to a validation experiment.

For a validation experiment, we therefore introduce a

for the validation data point ^{2}) with expectation _{vali},_{true}) and variance SD^{2}. Here, _{vali} denotes the design for the validation experiment. A validation confidence interval satisfying (11) allows a rejection of the model if a noisy validation measurement with error SD is outside the interval.

We now define the validation profile (log-)likelihood

with ^{∗ }=^{∗}(_{
θ
} LL(

Optimization of the likelihood (12) minimizes both, the mismatch of existing data RSS(^{
″
}satisfying the constraint optimization problem (9) considered for the prediction profile likelihood. It holds

i.e. the validation profile likelihood LL^{∗ }can be scaled to the prediction likelihood via

where ^{
′
}=_{vali},^{∗}(^{∗ }estimated from

Optimization with nonlinear constraints is a numerically challenging issue. Therefore, (16) provides a helpful way to omit constraint optimization. The VPL can be calculated with SD > 0 like a common least-squares minimization and is then afterwards rescaled to obtain the PCI for the true value.

Results

Small illustration model

First, a small but illustrative model of two consecutive reactions

with rates _{1 }= 0.05,_{2 }= 0.1 and initial conditions _{3 }= 1,

For the simulated measurements, Gaussian noise ^{2}) with

Panel (a) in Figure
^{∗} is shifted to zero in all figures. The threshold corresponding to the 90% confidence level is plotted as horizontal line. As explained in the Methods section, the projections to the horizontal axis yields the respective confidence intervals for a prediction or for a validation experiment. The constraint optimization procedure is infeasible for

Illustration model

**Illustration model.** The three figures in panel (**a**) show the dynamics and measurement realization for the small model used for illustration purpose. C(t) is measured and the dynamics of all states, i.e. A(t), B(t), and C(t), is intended to be predicted. Panel (**b**) shows as an example the prediction profile likelihood (gray dashed curve) and validation profile likelihood (black dashed curve) of A(t = 10). Thresholding yields confidence intervals for prediction (gray vertical lines) and validation (black vertical lines). The threshold and the respective projections correspond to the **c**)-(**e**) show prediction confidence intervals (gray) for the unobserved states A(t), B(t), as well as for the measured state C(t). The prediction profile likelihood functions are plotted as black curves in vertical direction. Non-observability is illustrated in panels (**f**)-(**h**). Panel (**f**) shows a realization of the measurements for a design which does not provide sufficient information about the steady state of C. This leads to a flat prediction profile likelihood for large values for A(t) as shown in panel (**g**), as well as for B(t) for t > 0 as plotted in panel (**h**). A flat prediction profile likelihood in turn yields unbounded prediction and validation confidence intervals and non-observability of A(t) and B(t) as indicated by the gray shaded regions.

The calculation of the prediction and validation confidence intervals has been repeated for

For plotting the confidence intervals along the time axis, the PCIs evaluated the eleven time points have been interconnected by cubic piecewise interpolation. The displayed confidence intervals constitute the propagation of information from the measurements of

In the Additional file

To illustrate the effect of

Panel (g) shows the prediction confidence intervals for

MAP kinase signaling model

Next, an ODE model of cellular signal transduction has been used to illustrate our method in a realistic setting. For this purpose, a model of the ^{∗}, Mek, Mek^{∗}, Mek^{∗∗}, Erk, Erk^{∗}, and Erk^{∗∗}which play a very prominent role in many cellular processes, e.g. in cell proliferation. A star ‘*’ denotes phosphorylation of the protein which biologically acts as activation.

Panel (a) in Figure

MAP kinase model

**MAP kinase model.** Panel (**a**) shows the MAP kinase model according to
**b**) the dynamics of the MAP kinase model as well as simulated data set are plotted. The 90% confidence intervals of the dynamic variables for predictions (dark gray) and for validation experiments (light gray) for this noise realization are plotted in panel (**c**). The size of the prediction confidence interval (PCI) is plotted as a dashed-dotted line. In absolute concentrations, the dynamics of Erk^{∗∗}has the largest PCI at t = 181 seconds, i.e. when the negative feedback is activated. Also, the dynamics of Mek^{∗}is only badly observable in our example. Measurements of both would be very informative for better calibrating the model.

It is assumed that the total amount of the phosphorylated forms for each protein, i.e. Raf^{∗}, the sum of Mek^{∗} and Mek^{∗∗} as well as the sum of Erk^{∗} and Erk^{∗∗}, are measured. This observational assumption holds for example for phospho-specific antibodies such as utilized for western blotting. The measurement times are set to 0,100,…,1000 seconds. Again, additive Gaussian noise is assumed. The standard deviation has been set to

In panel (b) of Figure

The prediction confidence intervals show how precisely the dynamics is inferred by the available data. The temporal behavior of Raf, Raf^{∗}is quite well determined, i.e. the size of the PCI is below 40 nM. Similarly, the unphosphorylated states of Mek and Erk have narrow prediction confidence intervals. For Mek^{∗} the concentration dynamics is only predicted within rather large intervals which for most time points nearly span a range between zero and 100 nM.

The largest absolute size of the prediction confidence interval of 176 nM is obtained for Erk^{∗∗} after 181 seconds. This is the point in time where the negative feedback is activated. Additional experimental investigation of this condition is very informative to further specify the dynamic behavior of the MAP kinase cascade in our example. Further considerations concerning experimental planning are provided in detail in the Additional file

Discussion

Existing approaches for prediction confidence intervals like MCMC

In this paper, we present a contrary procedure. The model prediction space is sampled directly and the corresponding model parameters are determined by constraint maximum likelihood to check the agreement of the predictions with the data. This concept yields the prediction profile likelihood which constitutes the propagation of uncertainty from experimental data to predictions.

If a comprehensive prior, i.e. for all parameters, would be available, a Bayesian procedure like MCMC where marginalization, i.e. integration over the nuisance dimensions is feasible could have superior performance. However, in cell biology applications, prior knowledge is very restricted because kinetic rates and concentrations are highly dependent on the cell type and biological context, e.g. on the cellular environment and biochemical state of a cell. Therefore, there is usually at most some prior information for few parameters available. Such prior information can be incorporated in our procedure without restricting its applicability by generalizing maximum likelihood estimation to maximum a-posterior estimation as discussed in the Additional file

In general, generating prediction confidence intervals given the uncertainty in a high-dimensional nonlinear parameter space requires large numerical efforts. However, this complication primarily originates from the complexity of the issue itself rather than from the methodological choice. In fact, the aim is approached by the prediction profile likelihood in a very efficient manner because scanning the parameter space by the constrained optimization procedure to explore the data-consistent predictions is more efficient than sampling parameter space without considering the predictions like it is performed for MCMC. Instead of sampling a high-dimensional parameters space, only the prediction space has to be explored for calculating a prediction profile likelihood, i.e. the optimization of the parameters reduces the high-dimensional sampling problem to exploring a single dimension.

The prediction confidence regions introduced above has to be interpreted

In contrast, if a single data set is utilized to generate many prediction intervals, e.g. predictions for several points in time as performed above, the results are

The prediction profile likelihood also provides a concept for experimental planning. Experimental conditions with a very narrow prediction confidence interval are very accurately specified by the available data. New measurements for such a condition on the one hand does not provide very much additional information to better calibrate the model parameters, and hence is from this point of view a bad choice for additional measurements. On the other hand, it very precisely predicts the model behavior under these certain conditions and is therefore a very powerful candidate setting for validating the model structure. Contrarily, large prediction confidence intervals indicate conditions which are weakly specified by the existing data and therefore constitute informative experimental designs for better calibrating the model. Because a design optimization on the basis of the prediction profile likelihood does not require any linearity approximation like common experimental design techniques, e.g. based on the

Another potential of the prediction profile likelihood shown in this article is its interpretation in terms of

Finally, it should be noted, that a prediction could be any function of the compounds and the parameters. In applications, e.g. a ratio of two compound concentrations is a characteristics of interest. In principle also integrals, peak positions and other functions of the dynamic states can be considered as predictions which could be targets for observability considerations as well as for the calculation of prediction and validation confidence intervals. This flexibility renders the prediction profile likelihood as a concept promising to resolve one bottleneck in computer-aided simulations of complex systems, the generation of reliable confidence intervals for predictions.

Conclusions

Computer-aided simulations are a well-established tool to study a system’s behavior. The applications range from forecasting climate changes

In this article, the prediction profile likelihood methodology is presented as a method for calculating the set of model predictions which are consistent with existing measurements without explicitly calculating the uncertainty of the parameters. This is performed numerically by constrained optimization and constitutes a powerful tool for assessing model predictions, performing observability analyses, and experimental design. The method is feasible for arbitrary dimensions of the parameter space. It only requires a proper calculation of the maximum likelihood value, i.e. a numerically reliable parameter optimization procedure. The task of sampling a high-dimensional parameter space reduces to scanning a one-dimensional prediction space. It therefore allows the calculation of confidence intervals for model predictions as well as confidence intervals for the outcome of validation experiments.

The applicability of the approach has been shown by a small but instructive system of two consecutive reactions and a published model for MAP kinase signaling. For the small system, it has been shown that the prediction profile likelihood yields desired coverage properties. Moreover, a setting inducing non-observability has been investigated which is characterized by unbounded prediction confidence intervals. For the MAP kinase model, prediction confidence intervals and validation confidence intervals for all dynamic states have been determined on the basis of measurements of the phosphorylated proteins. In addition, the applicability of the approach for experimental planning has been demonstrated.

Competing interests

The authors declare that they have no competing interests.

Authors’ contributions

CK developed the method, performed the simulations, and wrote major parts the manuscript. AR contributed to the establishment of the method and wrote parts of the manuscript. JT supported CK and AR in methodological issues and helped to draft the manuscript. All authors read and approved the final manuscript.

Acknowledgements

The authors thank our long-term experimental collaboration partners, especially Dr. Maria Bartolome-Rodriguez and Prof. Ursula Klingmüller and their groups for their support and their experience in practically relevant issues. In addition, the authors acknowledge financial support provided by the BMBF-grants 0315766-VirtualLiver, 0315415E-LungSys, and 0313921-FRISYS as well as SBCancer DKFZ V.2 by the Helmholtz Society. The article processing charge was funded by the German Research Foundation (DFG) and the Albert Ludwigs University Freiburg in the funding programme Open Access Publishing.