Mathematical Institute, University of Oxford, Oxford, UK

Department of Biosystems Science and Engineering, , ETH Zurich, 8092 Zürich, Switzerland

School of Engineering and Computing Sciences, Durham University, UK

Abstract

Background

Dynamic mathematical models in the form of systems of ordinary differential equations (ODEs) play an important role in systems biology. For any sufficiently complex model, the speed and accuracy of solving the ODEs by numerical integration is critical. This applies especially to systems identification problems where the parameter sensitivities must be integrated alongside the system variables. Although several very good general purpose ODE solvers exist, few of them compute the parameter sensitivities automatically.

Results

We present a novel integration algorithm that is based on second derivatives and contains other unique features such as improved error estimates. These features allow the integrator to take larger time steps than other methods. In practical applications, i.e. systems biology models of different sizes and behaviors, the method competes well with established integrators in solving the system equations, and it outperforms them significantly when local parameter sensitivities are evaluated. For ease-of-use, the solver is embedded in a framework that automatically generates the integrator input from an SBML description of the system of interest.

Conclusions

For future applications, comparatively ‘cheap’ parameter sensitivities will enable advances in solving large, otherwise computationally expensive parameter estimation and optimization problems. More generally, we argue that substantially better computational performance can be achieved by exploiting characteristics specific to the problem domain; elements of our methods such as the error estimation could find broader use in other, more general numerical algorithms.

Background

In systems biology, mathematical models often take the form of system of ordinary differential equations (ODEs). These are approximations of the underlying mechanisms such as enzyme-catalyzed biochemical reactions that are applicable when molecule numbers are sufficiently high, and when the spatial distributions of components in a cell can be neglected. More specifically, ODE models consider the rate of change in a set of states (e.g. species concentrations) as a function of the system’s current state, its inputs, and its inherent kinetic parameters that capture, for instance, affinities of molecular interactions

In contrast to systems modeling in domains such as physics, however, model parameters and initial conditions for systems biology models are often not known, or they can only be roughly approximated. As few kinetic parameters can be measured directly, parametric uncertainty often prevails

From a computational point of view it is important to note that in all but the simplest cases, starting from a set of initial conditions, there is no direct way to compute the solutions of a system of ODEs (states or parameter sensitivities in our case) for an arbitrary time. The variables are therefore integrated numerically in small steps over time, until the desired end time is reached. Consequently, efficient and accurate numerical integration methods are critical for many applications.

The computational effort for numerical integration is linked to the system size, and over time mathematical models have become increasingly detailed to achieve better predictions. Nevertheless, even models of moderate complexity result in numerical challenges when parameter sensitivities are needed. For instance, the parameter sensitivities can be integrated naively alongside the system variables, but this implies integrating a system of size _{
x
}×(1 + _{
p
}), where _{
x
} and _{
p
} are the number of system variables and system parameters respectively

Additionally, the solution of a system of ODEs is often used in system identification processes where global optimization or probabilistic inference are required

These issues are not new and they concern many application domains. Considerable efforts have been invested in establishing reliable and efficient general-purpose ODE solvers for dynamic systems and—to a lesser extent—for the associated parameter sensitivities. Here, however, we are concerned with solving systems of ODEs as they typically occur in the simulation of biochemical reaction networks in systems biology

Dynamic models of (bio)chemical networks

When the effects of stochastic noise and of discrete molecule numbers are negligible, ODE systems can be used to describe chemical or biological reaction networks. The
_{x} time-dependent state variables _{i
}(**p**), _{x
}, which represent the concentrations of the molecules of interest at time _{0}, evolve following

where **f**(**x**(**p**) is a system of functions
_{i
}(**x**(**p**) modelling the conversion rate of each respective variable _{i
}(**p**)
at time **p**
is a vector of _{p
}
system parameters.

The local parameter sensitivities with respect to some parameter _{k
} are defined as

which is the vector of the derivatives of all variables _{i
}
with respect to the parameter _{k
}. Similar to the dynamics in Eq. (1), the parameter sensitivities’ time evolution follows a system of ODEs given by differentiating Eq. (2) with respect to

where **J**_{f
}(**x**(**f**(**x**(**x**(**p**
to simplify notation. Initial conditions for Eq. (3) are set according to whether the initial conditions for the states in Eq. (1) depend on the parameters or not

Consider, for example, the biochemical scheme of a Michaelis-Menten type enzymatic reaction

where _{1−4}
correspond to enzyme, substrate, enzyme-substrate complex, and product concentrations, respectively. With mass-action kinetics, the reaction network translates to the dynamic system

Such problems are often well solved by general purpose ODE solvers, but (bio)chemical reaction networks offer a number of features that may be exploited by more specialized solvers, resulting in faster and/or more precise simulations. For instance, in enzyme kinetics, reversible association and dissociation processes are usually much faster than product formation. The resulting stiffness severely limits the types of numerical methods that can be used for ODE integration.

An opportunity for increasing solver efficiency, however, presents itself because most (bio)chemical reaction networks are only weakly interconnected. More specifically, the change in every concentration _{i
} usually depends on the concentration of very few other products. Poor connectivity is reflected in **J**_{f
}(**x**(

with closed and open circles indicating non-zero and zero elements, respectively. Even in this dense sub-network, the number of non-zeros

Many large-scale biological networks have a scale-free structure, that is, most of their nodes have few interactions, but a small number of hubs with many interactions exist

Two more general aspects also need to be considered. Firstly, due to the growing use of abstract modeling software, the reactions and the underlying reaction equations are usually available to us as abstract models, such as **s**_{k
}(

Methods for ODE integration

Almost all ODE integrators work under the assumption that the change in each variable _{
i
}
over time can be modeled using a polynomial in **x**(_{0} to advance the system by a step of size

If the factors ^{k
}**x**(^{k
} / **x**(**x**(**g**_{n
}(**x**(**g**
_{
n
}(**x**(**g**_{n
}(**x**(**x**(

Within the two larger classes, different integrators are characterized by the amount of previous values of **x**(**x**(

**Integrator**

**Nodes, explicit ^{1}**

**Nodes, implicit**

^{1}For multi-step methods, the index

Euler

**x**(

**x**(

Backward Differentiation Formula (BDF)

**x**(_{
k
})

**x**(**x**(_{
k
})

Adams-Moulton (AM)

**x**(

Second-derivative rule (this work)

—

**x**(

Despite the commensurate degree of freedom in designing ODE integrators, and the number of algorithms for the numerical integration of ODEs that have been published over the past 40 years, only very few of them have found wide-spread application. Practical considerations—any method should be easily accessible to its end users, who are usually not interested in manipulating or even formulating the underlying equations themselves—are certainly major causes for this convergence

In this area, the first major piece of software was the

Methods

A second-derivative integrator

All ODE solvers mentioned above use only values of **x**(**x**(

(for notational simplicity, we will write **J**_{f
}(**f**(**J**_{f
}(**x**(**f**(**x**(

The use of second derivatives was first suggested in

**Supplementary Text and Figures for A Specialized ODE Integrator for the Efficient Computation of Parameter Sensitivities**.

Click here for file

**Supplementary Source Code for A Specialized ODE Integrator for the Efficient Computation of Parameter Sensitivities**.

Click here for file

The second derivatives in Eq. (6) may seem somewhat clumsy and expensive to evaluate since they require the construction of the Jacobian **J**_{f
}(**J**_{f
}(**f**(**J**
_{
f
}(**J**_{f
}(**f**(**f**(

In most cases, the evaluation of the second derivatives is not much more expensive than the evaluation of **f**(

For our second-derivative integrator, we construct an interpolating polynomial **g**_{4}(**x**(**x**(

which, expanding **g**_{4}(

where the right-hand side is the polynomial through **x**(

where **M**(**J**_{Jf
}(**x**(

The Jacobians

Using a second-derivative scheme, the evaluation of each Newton iteration is roughly twice as expensive as for first-derivative methods of the same degree since, in addition to **f**(**J**_{f
}(**f**(**x**(

for the truncation error of our second-derivative formula. For first-derivative methods of the same degree, assuming a constant step size

in the case of the BDF and the Adams-Moulton formula of degree four, respectively. These truncation errors are 72 times and 19 times larger than the error of our second-derivative formula (assuming the fifth derivative **x**
^{(5)}(_{
n−3},_{
n + 1}], see Section S2 and Figure S1A in Additional file

Error estimates and step size adjustment

In any ODE integration scheme, the local error estimate and the step-size adjustment are crucial to both its accuracy and its efficiency. The step-size adjustment uses the error estimate of a previous integration step to predict the largest possible next step

In most implicit ODE solvers, the local error is either estimated from the difference between the initial estimate **x**(**x**(**x**(_{1}, we need to compute a better approximation of degree _{2} > _{1}. The difference between both converged solutions **x**_{1}(**x**_{2}(**x**
^{⋆}(

We propose a different approach that may better reconcile accuracy with computational cost. We first compute the converged lower-degree solution
**x**_{1}(**x**_{2}(

However, for our second-derivative solver, we can compute the second approximation as the polynomial **g**_{5}(**g**_{4}(**x**(_{−1}). In this case, the weights in the Newton iteration matrix are similar. If the current step size _{−1} are equal, the weights for **J**_{f
}(**J**_{Jf
}(**g**_{4}(**x**
_{1}(

Note that the estimate **x**
^{(5)}(

Note that if the assumptions on **x**
^{(5)}(

Parameter sensitivities

For _{x
} variables and _{p
}
parameters, the naive approach to sensitivity calculation implies integrating a system of _{x
}×(1 + _{p
}) variables and, by consequence, inverting or decomposing matrices of that size within the Newton iteration. However, the system variables **x**(**x**(**x**(**s**_{k
}(

To integrate the parameter sensitivities in our second-derivative solver in a similar way, we need to compute the second derivatives

The equation in the implicit step using the second-derivative rule in Eq. (7) for the parameter sensitivities **s**
_{
k
}(

which, after isolating the sole unknown term **s**
_{
k
}(

We then have two alternatives to compute **s**_{k
}(**x**(**x**(**s**
_{
k
}(_{x
}, however, this extra matrix computation may be advantageous over running the Newton iteration for each parameter _{k
}.

Furthermore, if the Jacobians do not vary significantly over time, they can be re-used as the matrices _{p
} Newton iterations to compute the parameter sensitivities iteratively outweighs the cost incurred by the slower convergence due to using older Jacobians in the next step. In our second-derivative integrator, we therefore compute the parameter sensitivities directly as per Eq. (15).

Framework for conversion of SBML models

In order to generate the matrices **J**_{f
}(**J**_{Jf
}(**f**(**p**
and

Results and discussion

Implementation and testing

We implemented the second-derivative ODE integrator as _{mex}
_{c}

We compared our algorithm against three integrators which use Newton’s method to compute each implicit step:

1.Matlab’s default integrator for stiff systems,

2.the

3.the

The Matlab interface supplied by the

For performance evaluation, we selected a number of curated systems biology models from the BioModels database _{
f
}).

**Model**

**
n _{x
}**

**
n _{p
}**

**
nnz(J _{f
})**

**
nnz(J _{Jf
})**

**
t
**

**Comments**

^{1}Abbreviations used: ERK = extracellular regulated kinase; EGF = epidermal growth factor; IL-6 = interleukin 6; FGF = fibroblast growth factor.

Hornberg

8

19

22

32

[0,100]

Steady state, ERK^{1} phosphorylation and kinase/phosphatase control.

Kholodenko

23

51

137

372

[0,100]

Steady state, short term signaling by the EGF receptor.

Singh

66

109

323

846

[0,35 000]

Steady state, IL-6 signal transduction in hepatocytes.

Borisov

90

136

468

2156

[0,1 000]

Steady state, insulin-EGF network interactions in mitogenic signaling.

Ung

200

314

956

4038

[0,4 000]

Steady state, regulation of EGFR endocytosis and EGFR-ERK signaling by crosstalk.

Elowitz & Leibler

6

8

12

18

[0,1 000]

Oscillatory, synthetic network of transcriptional regulators.

Leloup & Goldbeter

10

48

30

52

[0,300]

Oscillatory, circadian oscillations of PER and TIM proteins in

Wolf

13

40

47

99

[0,300]

Oscillatory, autonomous metabolic oscillations in continuous culture of

Goldbeter & Pourquié

20

73

47

76

[0,300]

Oscillatory, segmentation clock by crosstalk of Notch, Wnt, and FGF pathways.

Xie & Kulasiri

24

49

57

107

[0,100]

Oscillatory, circadian rhythms in

Integrator performance without parameter sensitivities

The results of the performance comparison without sensitivity analysis for a wide range of integration tolerances are summarized in Figure _{mex}

Performance comparison without parameter sensitivities

**Performance comparison without parameter sensitivities.** Performance comparison for integration of ODE-based systems biology models without parameter sensitivities. (**A**) Computation times, (**B**) number of integration steps, (**C**) number of r.h.s. evaluations **f**(**x**), and (**D**) number of evaluations of the Jacobian **J**_{f}(**x**)
as a function of the relative numerical tolerance. Symbols specify the integrators _{mex}**E**) Computation times and (**F**) number of integration steps as a function of numerical precision (see main text for definition) in analogy to (**A**) and (**B**).

To assess the relative accuracy of ^{−15}, analogously to the precision/work tests in

Precision/work diagrams without parameter sensitivities

**Precision/work diagrams without parameter sensitivities.** Precision-work diagrams for integration without parameter sensitivities. (**A-J**) Computation times for the individual models (see X-axis for model specifications) as a function of precision using _{mex}

Integrator performance with parameter sensitivities

The performance comparison with sensitivity calculations requires two additional considerations: Since _{
x
}×(_{
p
} + 1)
including an analytic sparse Jacobian for each model, and the sensitivities **s**
_{
k
}(

In all cases, parameter sensitivities were integrated to the same precision as the system variables. As with the integration without sensitivities, precision/work diagrams were computed for all models with sensitivities, omitting the cases in which

As one detailed example, Figure ^{−6}. In most cases, the compute times with sensitivities are substantial (see also Additional file **f**(·) is evaluated, the algorithm also computes _{
p
}, which, as per Eq. (3), requires an evaluation of the Jacobian **J**
_{
f
}(·). As a consequence, our second-derivative integrator outperforms the Sundials solver when implemented in C using the Matlab _{mex}
_{mex}

Performance comparison with parameter sensitivities

**Performance comparison with parameter sensitivities.** Performance comparison with parameter sensitivities. (**A**) Computation times for the individual models listed in Table ^{−6}
using _{mex}**B**) CPU times with sensitivity calculation as in (**A**) relative to CPU times without sensitivity calculation. (**C**) Average, normalized (see below) CPU times with sensitivities as a function of the relative numerical tolerance for _{mex}**D**) Relative numbers of integration steps (open black circles), of function evaluations **f**(**x**)
(filled black circles), and of evaluations of the Jacobians **J**_{·}(·)
(open red squares) for _{mex}**C**, **D**) are normalized to the corresponding measures for

In order to obtain results independent of any potential inefficiencies of the Matlab interface, the same performance analysis was run using _{c}

Performance comparison of C-language integrators with parameter sensitivities

**Performance comparison of C-language integrators with parameter sensitivities.** Performance comparison with parameter sensitivities of the C-language version _{c}**A**) Computation times for the individual models listed in Table ^{−6}
using **B**) Average, normalized (see below) CPU times as a function of the relative numerical tolerance for **B**) are normalized to the corresponding measures for

The higher efficiency of _{mex}
_{c}

Precision/work diagrams with parameter sensitivities

**Precision/work diagrams with parameter sensitivities.** Precision-work diagrams for integration with parameter sensitivities. (**A-G**) Computation times for all models for which the systems dynamics were solved with all ODE integrators (see X-axis for model specifications) as a function of precision using _{mex}

To explain the performance, consider that although _{mex}
**f**(·) and of the Jacobians **J**
_{(·)}(·)
are needed (Figure

Conclusions

We have presented an integrator for ODE systems resulting from the modeling of chemical and biological reaction networks, which are often stiff and sparse. For the realistic systems biology models tested, the new integrator outperforms commonly used state of the art integrators when parameter sensitivities are required. It is competitive in integrating the system equations alone, despite limitations for specific models near the steady state. The improvements with respect to sensitivity calculations are critical for many applications to drive highly compute-intensive (global) optimization and estimation processes.

The improvements themselves are due to a combination of several factors: The more accurate second-derivative rule allows us, in combination with a better error estimate, to take larger steps, which in turn allows us to reduce the number of otherwise expensive sensitivity calculations. The re-use of the Jacobians from the sensitivity calculations further reduces the total computational costs. Although each integration step is more expensive than in first-derivative methods, due to the additional second-derivative information that needs to be computed, far less steps are required in total, resulting in a more efficient method.

To be of practical relevance for applications in systems biology, _{mex}
^{a}, and we plan to make them more easily available through integrated modeling environments such as the SBToolbox2

In terms of numerical algorithms, to our knowledge, this is the first practical application of a second-derivative integration method with good performance. Key, novel features of our integrator, such as a more precise error estimator and direct computation of parameter sensitivities with re-use of the Jacobians, may well be suited for other problems or types of integrators. Importantly, while our integrator has been developed for (bio)chemical and reaction networks, it is still quite general in targeting stiff and sparse ODE systems. Overall, we feel that a lot is to be gained by adapting general algorithms to specific problem domains, and that results from the work on specific problems can spill over to the broader field.

Availability

The integrator (Matlab and C-language versions) and the model conversion framework are available via

Endnotes

^{a}We note that it is not possible to execute _{mex}

Competing interests

The authors declare that they have no competing interests.

Authors’ contributions

PG developed the second-derivative integration scheme for the parameter sensitivities and the new error estimate, as well as the final Matlab,

Acknowledgements

We thank Ernst Hairer for comments and discussions. This work was supported by the EU FP6 project BaSysBio (LSHG-CT-2006-037469), the EU FP7 project UniCellSys (HEALTH-F4-2008-201142), the Swiss Initiative for Systems Biology SystemsX.ch (RTD project YeastX), by an ETH Excellence Scholarship (LW), and by Swiss National Science Foundation’s Individual Support Fellowships Nr. PBEZP2-127959 and Nr. PA00P2-134146 (PG).