Email updates

Keep up to date with the latest news and content from BMC Medicine and BioMed Central.

Journal App

google play app store
Open Access Research article

Validation of a model to investigate the effects of modifying cardiovascular disease (CVD) risk factors on the burden of CVD: the rotterdam ischemic heart disease and stroke computer simulation (RISC) model

Bob JH van Kempen12, Bart S Ferket12, Albert Hofman1, Ewout W Steyerberg3, Ersen B Colkesen6, S Matthijs Boekholdt5, Nicholas J Wareham7, Kay-Tee Khaw8 and MG Myriam Hunink124*

Author affiliations

1 Department of Epidemiology, Erasmus MC Rotterdam, dr Molewaterplein 50, Rotterdam, 3015 GE, the Netherlands

2 Department of Radiology, Erasmus MC Rotterdam, dr Molewaterplein 50, Rotterdam, 3015 GE, the Netherlands

3 Department of Public Health, Erasmus MC Rotterdam, dr Molewaterplein 50, Rotterdam, 3015 GE, the Netherlands

4 Department of Health Policy and Management, Harvard School of Public Health, 677 Huntington Avenue, Boston, MA 02115, USA

5 Department of Cardiology, Amsterdam Medical Center, Meibergdreef 9, Amsterdam, 1150 AZ, the Netherlands

6 Department of Cardiology, Antonius Hospital, Koekoekslaan 1, Nieuwegein, 3435 CM, the Netherlands

7 Medical Research Council Epidemiology Unit, Hills Road, Cambridge, CB2 0QQ, UK

8 Department of Public Health and Primary Care, University of Cambridge, Robinson Way, Cambridge, CB2 0SR, UK

For all author emails, please log on.

Citation and License

BMC Medicine 2012, 10:158  doi:10.1186/1741-7015-10-158

The electronic version of this article is the complete one and can be found online at: http://www.biomedcentral.com/1741-7015/10/158


Received:24 April 2012
Accepted:6 December 2012
Published:6 December 2012

© 2012 van Kempen et al; licensee BioMed Central Ltd.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Background

We developed a Monte Carlo Markov model designed to investigate the effects of modifying cardiovascular disease (CVD) risk factors on the burden of CVD. Internal, predictive, and external validity of the model have not yet been established.

Methods

The Rotterdam Ischemic Heart Disease and Stroke Computer Simulation (RISC) model was developed using data covering 5 years of follow-up from the Rotterdam Study. To prove 1) internal and 2) predictive validity, the incidences of coronary heart disease (CHD), stroke, CVD death, and non-CVD death simulated by the model over a 13-year period were compared with those recorded for 3,478 participants in the Rotterdam Study with at least 13 years of follow-up. 3) External validity was verified using 10 years of follow-up data from the European Prospective Investigation of Cancer (EPIC)-Norfolk study of 25,492 participants, for whom CVD and non-CVD mortality was compared.

Results

At year 5, the observed incidences (with simulated incidences in brackets) of CHD, stroke, and CVD and non-CVD mortality for the 3,478 Rotterdam Study participants were 5.30% (4.68%), 3.60% (3.23%), 4.70% (4.80%), and 7.50% (7.96%), respectively. At year 13, these percentages were 10.60% (10.91%), 9.90% (9.13%), 14.20% (15.12%), and 24.30% (23.42%). After recalibrating the model for the EPIC-Norfolk population, the 10-year observed (simulated) incidences of CVD and non-CVD mortality were 3.70% (4.95%) and 6.50% (6.29%). All observed incidences fell well within the 95% credibility intervals of the simulated incidences.

Conclusions

We have confirmed the internal, predictive, and external validity of the RISC model. These findings provide a basis for analyzing the effects of modifying cardiovascular disease risk factors on the burden of CVD with the RISC model.

Keywords:
Cardiovascular disease prevention; Simulation modeling; Model validation

Background

Decision models are being increasingly used to guide decisions on medical interventions in healthcare [1-3]. Both for healthcare policy-makers who have to make decisions for specific populations and weigh both benefits and costs, and for a general practitioner facing a medical decision for a particular patient, decision models can provide valuable information to aid the decision at hand. Empirical and trial-based studies on (cost-)effectiveness of medical interventions often evaluate a limited number of strategies, and typically cover a limited period of follow-up. Decision modeling can overcome these limitations by synthesizing the available information and extrapolating short-term study results, providing policy-makers with information on expected long-term outcomes and accompanying uncertainties [4]. However, because decision models are based on a necessarily simplified representation of the underlying disease and the intervention being studied, the validity of the model is not automatically guaranteed. Earlier research has shown that importance of model validation before the results of a simulation study can be used for medical decisions [5-8].

Three types of validity have been described. With internal validation, the output of the model is compared with the data that was used to build the model [9,10]. Although model output and data are inherently dependent on each other with this type of validation, internal validity is a necessary condition, and provides an indication of how well the model output represents the data. Whereas the follow-up period in observational studies and clinical trials is necessarily limited, medical decisions often require long-term outcomes. A common approach is to extrapolate the results of a simulation model beyond the period on which it was originally based. The validity of a model with regard to its accuracy to simulate results beyond the original timeframe is called 'predictive' or 'prospective' validity [11,12], and constitutes the second form of validity. In evaluating predictive validity, the model output is compared with data from the new follow-up period, which has become available after the model was developed. The extent to which the results of a model can be applied to other populations different from the original one is the third form of validity, external validity [9,10]. Because potential differences between populations affect many of the parameters used in a model, external validity is a more rigorous test of model validity than the other two validity measurements.

The objective of this study was to assess the internal, predictive, and external validity of the Rotterdam Ischemic Heart Disease and Stroke Computer Simulation (RISC) model [13]. The RISC model was designed to investigate the effects of modifying cardiovascular disease (CVD) risk factors on the CVD burden in a general population. The model is based on data from the Rotterdam Study, a cohort follow-up study of 7,983 adults aged 55 years and older. Validation of the RISC model is required before the results produced by the model can be used for decision-making.

Methods

The model

The RISC model is a Monte Carlo state-transition model (schematically presented in Figure 1) with six states: 1) the CVD death state, 2) the non-CVD death state, 3) the coronary heart disease (CHD) state, 4) the stroke state, 5) the CHD and stroke state, and 6) the well state (being alive without CHD or stroke). The model simulates incident CVD events in individuals with and without previous CVD based on risk-actor-dependent transition probabilities, using Cox regression equations.

thumbnailFigure 1. Schematic presentation of the Rotterdam Ischemic Heart Disease and Stroke Computer Simulation (RISC) model. CHD, coronary heart disease; CVD, cardiovascular disease. Arrows indicate transitions between the health states.

Individual risk-factor profiles are modeled and tracked over time. Incident CVD events are counted using tracker variables during the period of simulation. CHD is defined as: acute myocardial infarction (International Classification of Diseases, 10th edition (ICD-10) code I21), percutaneous transluminal coronary angioplasty (PTCA) and coronary artery bypass graft (CABG). Stroke is limited to non-hemorrhagic and unspecified strokes (ICD-10 codes I63, I64). Cardiovascular death is defined as mortality due to hypertensive diseases (ICD-10 codes I10 to 15), ischemic heart disease (ICD-10 codes I20 to I25), sudden cardiac death (ICD-10 codes I46, I49), congestive heart failure (ICD-10 code I50), cerebrovascular disease (ICD-10 codes 160 to 167), other arterial disease (ICD-10 codes I70 to I79), or sudden death (ICD-10 code R96). Non-cardiovascular death is defined as mortality due to all other causes (all other ICD-10 codes). The model was built using TreeAge software (version Data Professional release 2009; TreeAge Software, Inc., Williamstown, USA). Detailed information about the model has been given in an earlier publication [13] (see also Additional file 1).

Additional file 1. Technical appendix. Technical appendix with additional information on the Rotterdam Ischemic Heart Disease and Stroke Computer Simulation (RISC) model and analyses.

Format: DOC Size: 238KB Download file

This file can be viewed with: Microsoft Word ViewerOpen Data

Ethics approval

In the RISC model, the risk-factor profiles and transition probability functions were based on data from the Rotterdam Study population. The Rotterdam Study was originally approved by the institutional review board of the Erasmus Medical Center and by the review board of The Netherlands Ministry of Health, Welfare and Sports [14].

Data sources

This population consisted of 7,983 respondents from a random sample of adults aged 55 years and older, who were recruited between 1990 and 1993 and were residing in Ommoord, the Netherlands. Of these 7,983 respondents, 6,871 both visited the research center and signed an informed consent document. These individuals were followed up from 1990 to 2000; the follow-up consisted of three physical examinations with interviews, and the surveillance of hospital admissions, death registries, and other available medical sources ensured accurate follow-up of death and clinical manifestations of CVD.

In 3,501 of the participants, all important characteristics for prediction of CVD were known, and the RISC model is based on 5-year follow-up data from these 3,501 individuals. The risk factors considered for the transition probability functions were age, sex, smoking status, systolic and diastolic blood pressure, body mass index, waist-to-hip ratio, ankle-brachial index; levels of plasma glucose, plasma total cholesterol, high-density lipoprotein (HDL) cholesterol, and plasma creatinine; family history of CVD, presence of hypertension (blood pressure over 160/90 or use of anti-hypertensive medication) or diabetes mellitus; manifestations of intermittent claudication, angina pectoris, atrial fibrillation or transient ischemic attacks; and prevalent CVD. Details about the assessment of these risk indicators have been described in earlier publications [15]. The Cox regression equations that described the state-transition probabilities were centered around the mean of the risk factors of these 3,501 participants. This enabled the analysis of populations other than the original one, by substituting the centered cumulative baseline hazard and the average values of the risk factors by the values from the other population(s).

Simulation of parameter uncertainty

The RISC model allows for the evaluation of parameter uncertainty [16]. The majority of the parameter uncertainty in the model stems from the β-coefficients underlying the transition probability functions, and these β-coefficients are potentially dependent on each other. To model the uncertainty of the coefficients, 100 bootstrap samples of the study population were drawn. All the transition probability functions were fitted for every bootstrap sample, resulting in 100 sets of linked transition probability functions, which allowed for the dependency between them. The transition probabilities were based on Cox regression equations, and parameter uncertainty around the baseline hazards of the CVD events, CVD death, and non-CVD death was also included.

Simulation of heterogeneity

The RISC model was designed to simulate individuals who each had a unique risk-factor profile for CVD [17]. Model outcomes are expected to be different for individuals with high-risk profiles (older age, male, high blood pressure, high lipid levels, diabetes mellitus) than for those with more favorable profiles. To allow for differences in outcomes resulting from individual differences in risk-factor profiles (that is, heterogeneity), we used the RISC model to simulate different individuals one at a time.

Simulation of the history for each individual

The risk factors used in the RISC model reveal trends over time. As an example, total cholesterol levels were found decline with age in the Rotterdam Study. To take these trends in risk factors over time into account, each risk-factor profile for a particular individual was updated every 5 years during their simulated life in the model, based on the trends seen during the first 5 years in the Rotterdam Study. Therefore, the development of the risk factors needed to be tracked over time.

Events occurring during an individual's simulated life could influence the occurrence of other events. As an example, a CHD event increases the risk of dying in subsequent years. All cardiovascular events in the RISC model were therefore tracked and linked to the transition probabilities. The inclusion of variables used to track CVD events and changes in risk factors over time for each individual required the simulation of each individual multiple times to account for stochastic uncertainty [17].

Internal and predictive validation

From our cohort of 3,501 individuals from the Rotterdam Study on which the RISC model was based, we selected 3,478 who had at least 13 years of follow-up as of 1 January 2007. The remaining subjects were lost to follow-up because they had moved out of the area or had discontinued their participation. We calculated the cumulative incidences for total mortality, CVD mortality, non-CVD mortality, CHD, and stroke as defined previously for the 13-year period of follow-up (beginning of year 1 until end of year 13). We then compared this with the simulated cumulative incidences of the same events during the 1st year until the end of the 13th year by the RISC model. We furthermore stratified the analyses for the internal and predictive validity for CVD mortality by tertiles of age for the 3,501 participants, and for men and women separately. We choose CVD mortality because it is one of the most important clinical outcomes, and there would be enough events for it in each stratum to obtain stable results.

External validation

For the external validation, we used data from the EPIC-Norfolk study [18], which is a prospective population study of 25,663 men and women aged 45 to 79 years old residing in Norfolk, UK. This study had been approved by the Norwich District Health Authority ethics committee, and all participants gave signed informed consent [18]. Participants were originally recruited from age and gender registers of general practices in Norfolk as part of the 10-country collaborative EPIC study designed to investigate dietary and other determinants of cancer. Additionally, characteristics including anthropometry, blood pressure, and lipid levels were obtained for the assessment of determinants of other diseases. For the baseline survey from 1993 to 1997, participants completed a detailed health and lifestyle questionnaire and attended a clinic visit. All participants were followed up and mortality, linked to the UK Office of National Statistics, was recorded. Participants admitted to hospital were identified by their unique National Health Service number by data linkage with the East Norfolk Health Authority (ENCORE) database, which identifies all hospital contacts throughout England and Wales for Norfolk residents.

The EPIC data did not contain all variables used in the RISC model. In particular, the following information was not readily available: ankle-brachial index, serum glucose levels, and a history at baseline of angina pectoris, atrial fibrillation, intermittent claudication, or transient ischemic attack. Consequently, we imputed the missing data in the EPIC dataset based on the multiple variables that were available [19]. All major risk factors such as age, sex, cholesterol levels, and blood pressure were available and did not need to be imputed.

We used EPIC-Norfolk mortality data from 1993 until 31 March 2008. From the 25,663 participants, we selected 25,492 who had a follow-up of at least 10 years. For the external validation, we calculated the cumulative incidence of CVD and non-CVD mortality in the EPIC dataset. We compared this with the simulated cumulative incidences of the same events after year 1 until year 10 by the RISC model, using the 25,492 EPIC profiles as input.

We did not calculate or simulate CHD and stroke events in the external validation, because the EPIC study did not document CABG and PCI events and furthermore, non-fatal events were only recorded if the patient was hospitalized. In the Rotterdam Study, both CABG and PCI were counted as CHD events, and all CHD and stroke events were recorded whether or not the patient was hospitalized, making the definition of CHD and stroke inherently different between the two cohorts [20,21].

Statistical analysis

Important baseline characteristics for the baseline 3,478 Rotterdam Study participants and 25,492 EPIC participants were calculated and tabulated to evaluate their differences.

To take into account parameter uncertainty, the heterogeneity of the participants, and the stochastic uncertainty, we performed a three-level simulation [16,17]. We calculated the mean and distribution around the mean of the cumulative incidences by drawing from 100 second-order sets of linked β-coefficients from the state-transition probabilities and values for the baseline hazards of the events (outer simulation loop for parameter uncertainty). For each set of linked β-coefficients and baseline hazards, we consecutively simulated 2,000 randomly drawn risk-factor profiles from the 3,478 Rotterdam profiles for the internal and predictive validation, and 2,000 from the 25,492 EPIC profiles for the external validation (middle simulation loop for heterogeneity). For each profile, 200 random walks were simulated, needed for the tracking of the individual cardiovascular histories (microsimulation, inner simulation loop for stochastic uncertainty). This implies 100 × 2,000 × 200 runs per analysis. We did not model any particular intervention or treatment in this study; only the observed history (current practice) was simulated for purposes of validation. For the stratified analyses we aggregated on the individual level (n = 3,501 × 200 × 100 runs per analysis).

For the internal and predictive validation, we determined the average simulated cumulative incidences of CVD death, non-CVD death, CHD, and stroke for the 13-year period. For the external validation, we determined the average simulated cumulative incidences of CVD death and non-CVD death for year 1 until year 13. Because the Rotterdam Study and EPIC-Norfolk population are potentially different with respect to the distribution of risk factors and incidence of CVD, we subsequently recalibrated the RISC model by substituting the centered cumulative baseline hazards and mean values of the risk factors from the original model based on the Rotterdam data with the corresponding ones from the EPIC-Norfolk cohort [22]. We then ran again 2,000 randomly drawn participants from the 25,492 EPIC participants.

For all cumulative incidences, we calculated the 2.5% and 97.5% percentiles of the variation around the average incidences (credibility intervals) from the RISC simulations, to quantify the influence of parameter uncertainty. We compared the observed with the simulated incidences for all events.

Results

Compared with the Rotterdam Study, the the EPIC-Norfolk study participants were 10 years younger on average, and there were more men in the EPIC-Norfolk study (Table 1). On average, EPIC participants had lower total cholesterol levels and higher HDL levels (Table 1). The number of Rotterdam Study participants with a history of CVD at baseline exceeded that of the EPIC participants.

Table 1. Baseline characteristics of the risk factors used in the Rotterdam Ischemic Heart Disease and Stroke Computer Simulation (RISC) model for the 3,478 Rotterdam study participants and 25,492 European Prospective Investigation of Cancer (EPIC)-Norfolk study participants

Internal and predictive validation

During the 13 years of follow-up, 367 CHD events, 343 stroke events, 494 CVD deaths, and 846 non-CVD deaths occurred in the 3,478 Rotterdam Study participants, The cumulative incidences of CVD and non-CVD mortality during the13 years of follow-up for the Rotterdam Study participants were compared with the incidences generated by the RISC model (Figure 2, Figure 3). The observed values, both during the first 5 years (internal validation) and for the extrapolated period (predictive validation), were consistent with the simulated ones. The cumulative incidences of CHD and stroke events during the 13-year follow-up were compared with the incidences generated by the RISC model (Figure 4, Figure 5). The observed values were again consistent with the simulated events. For the cumulative incidences of CVD mortality, stratified by tertiles of age, for men and women respectively, the observed values were also consistent with the simulated values (see Additional file 1, Figure S2, Figure S3).

thumbnailFigure 2. Cardiovascular disease (CVD) mortality during 13 years of follow-up. The first 5 years refer to the internal validation, the remaining years to the predictive validation. Simulated versus observed values for the Rotterdam Study data.

thumbnailFigure 3. Non-cardiovascular disease (CVD) mortality during 13 years of follow-up. The first 5 years refer to the internal validation, the remaining years to the predictive validation. Simulated versus observed values for the Rotterdam Study data.

thumbnailFigure 4. Coronary heart disease (CHD) events during 13 years of follow-up. The first 5 years refer to the internal validation, the remaining years to the predictive validation. Simulated versus observed values for the Rotterdam Study data.

thumbnailFigure 5. Stroke events during 13 years of follow-up. The first 5 years refer to the internal validation, the remaining years to the predictive validation. Simulated versus observed values for the Rotterdam Study data.

External validation and recalibration

During the 10-year follow-up of the 25,492 EPIC-Norfolk participants, 943 CVD deaths and 1,661 non-CVD deaths occurred. The cumulative incidence of CVD and non-CVD mortality during the 10-year follow-up of the 25,492 EPIC participants were compared with the incidences generated by the RISC model, using the EPIC-Norfolk profiles as input (Figure 6, Figure 7). The observed values were within the 95% credibility intervals of the simulated values, but the RISC model overestimated the incidences for all years, for both CVD and non-CVD mortality. We then estimated the cumulative incidences of CVD and non-CVD mortality, after substituting the centered cumulative baseline hazards and average values of the risk factors with those based on the EPIC data, which recalibrated the model (Figure 8, Figure 9). After this recalibration, the observed CVD and non-CVD mortality incidences matched the simulated incidences from the RISC model.

thumbnailFigure 6. Cardiovascular disease (CVD) mortality during 10 years of follow-up. Simulated versus observed values for the European Prospective Investigation of Cancer (EPIC)-Norfolk data.

thumbnailFigure 7. Non-cardiovascular disease (CVD) mortality during 10 years of follow-up. Simulated versus observed values for the European Prospective Investigation of Cancer (EPIC)-Norfolk data.

thumbnailFigure 8. Cardiovascular disease (CVD) mortality during 10 years of follow-up in the recalibrated model. Simulated versus observed values for the European Prospective Investigation of Cancer (EPIC)-Norfolk data.

thumbnailFigure 9. Non-cardiovascular disease (CVD) mortality during 10 years of follow-up in the recalibrated model. Simulated versus observed values for the European Prospective Investigation of Cancer (EPIC)-Norfolk data.

Discussion

In this study, we evaluated the internal, predictive, and external validity of the RISC model. The simulated cumulative incidence of CVD and non-CVD deaths, CHD events, and strokes adequately represented the data during the original follow-up period of 5 years on which the RISC model was based. Extrapolation of the simulated results beyond this period proved to be valid for 13 years of follow-up, the maximum length that we analyzed in this paper. Although the results of the RISC model overestimated the CVD and non-CVD mortality compared with the observed 10-year incidences in the EPIC-Norfolk population, recalibrating the model with the cumulative baseline hazards and mean values of the risk factors substantially improved performance.

Other decision models used to evaluate preventive and treatment strategies for CVD have been well established. A recent review by Unal et al. identified forty-two such models, of which six major ones have been described in detail [23]. Although some of the forty-two models reported assessment of validity, most did not. Of the six major models, three have not been validated [24-26], two models had information on internal validity reported [27,28], and an external validation had been performed fo two models [29,30].

In the present study, the predictive validity of the RISC model was tested against follow-up data for more than twice the length of the period on which the model was originally based. The fact that the observed and simulated incidences matched closely even when extrapolated beyond the original data makes it plausible to expect projections beyond 13 years to be valid as well. The trends in risk factors over time and their effects on the incidence of events, which are jointly modeled in the RISC model, seem to provide a valid basis to extrapolate results, without the need to recalibrate the model for the Rotterdam Study population. We furthermore showed the robustness of the internal and predictive validity by providing results for the stratified analyses by tertiles of age and sex. As for the external validation, the EPIC-Norfolk population was on average younger and healthier than the Rotterdam Study population. It was to be expected that an unadjusted model, using the baseline hazards and mean of the risk factors from the Rotterdam Study, would overestimate the observed incidences in the EPIC-Norfolk study. In the recalibrated model, we updated only the baseline cumulative hazards of the events and the mean values of the risk factors, a method very commonly used when applying models to other populations than that for which the model was originally developed in [22,31]. This result suggests that the relative strengths of the associations of the risk factors with the incidence of the events in the RISC model are the same for both the EPIC-Norfolk population and Rotterdam Study. The resulting external validity of the RISC model after this adjustment strongly supports this assumption.

Our analysis does have some limitations. The RISC model was designed to investigate the effects of modifying cardiovascular risk factors on the burden of CVD in the middle-aged and older general population. We validated the model in the EPIC-Norfolk data, which included people aged from 45 years upwards. Although most current guidelines on the primary prevention of CVD mostly start at the age of 45 years and older, some do (or in the future potentially will), suggest that CVD prevention should begin at an earlier age Whether the RISC model also performs well in a younger population remains to be determined. The RISC model is intended to be used for projections during the remaining lifetime of an individual. The model proved to be valid for projections during 13 years of follow-up, and for most older people this is sufficiently long to cover their remaining lifespan. For younger people, this is less likely, and model extrapolation beyond this period therefore has to be made, which currently has not been validated. Because the Rotterdam Study is ongoing, and longer follow-up data are being collected, we will be able to test whether this additional extrapolation is valid as well.

A number of risk factors used for the RISC model were not documented in the EPIC-Norfolk study. To make the EPIC-Norfolk dataset suitable for the RISC model, we imputed missing data based on the correlations between the missing risk factors and the documented variables. These correlations stemmed from the Rotterdam Study data, thereby introducing dependency between the (imputed) EPIC-Norfolk data and the RISC model. However, the major traditional risk factors such as age, sex, cholesterol level, and blood pressure were available in EPIC. The prevalence of a number of missing risk factors such as atrial fibrillation and intermittent claudication were low in the Rotterdam Study data on which the RISC model was developed, and the incremental value beyond the traditional risk factors of the other variables, such as the ankle-brachial index, has been found to be limited [32]. It is therefore less likely that the imputation influenced the external validity in favor of concordance. Although the EPIC-Norfolk dataset contains information on (hospitalized) patients with MI, the RISC model simulates CHD as a combined endpoint, including CABG and PCI. This is consistent with most clinical trials using similar combined endpoints. The design of the RISC model therefore did not allow for direct comparison of simulated MIs as a sole endpoint. Although acute MI is the major component of CHD, both CABG and PCI interventions are inherently different from acute MIs, and we therefore did not externally validate CHD events in the EPIC dataset.

At the time of this paper, we did not have datasets other than EPIC-Norfolk at our disposal to perform additional external validation. The fact that the RISC model, after updating the model with the baseline hazards and mean values for the risk factors from EPIC, proved to be valid for the EPIC-Norfolk cohort, does not automatically imply that it will be valid in other populations as well. The EPIC-Norfolk cohort was younger on average, and included more men than the RISC cohort. However, the fact that the cohort was different with regard to these important risk factors, and yet RISC still provided valid results, does make a strong case that the model will be valid for other cohorts as well. We do intend to validate the model with other data as they become available. Both the Rotterdam Study and the EPIC-Norfolk study were population-based studies and included individuals regardless of pre-existing risk-factor profiles or disease status. Although risk-factor distributions of the study participants might in principle be different from the populations they intend to represent, it is very likely that the RISC model is valid for most western European populations in general after adjusting for baseline hazards. A simpler model with a reduced set of parameters, excluding the less common ones such as atrial fibrillation and ankle-brachial index, would possibly allow for a more rapid validation process in other populations. In an ongoing effort to optimize our model, we also intend to make efforts to simplify our current model.

We modeled and validated the cardiovascular histories of the participants of the Rotterdam Study and EPIC-Norfolk cohort as they were observed; that is, without any interventions. Although the results with regard to this validity seem promising, the RISC model will be used to evaluate interventions for the primary prevention of CVD. In that case, the validity of the model to evaluate an intervention depends not only on the observed CVD history, but also on the extent to which other structural assumptions are made, such as modeling the treatment effect of an intervention [33]. A more extensive framework of model validation proposed by Kopec et al. [34] also includes between-model comparisons, and comparisons of evidence from examining the consequences of model-based decisions. Between-model comparisons are specifically useful when analyzing certain interventions compared with the natural history of the disease, as we did in the current analysis. Being a simplifying abstraction of reality, a model will be valid with regard to some (but not necessarily all) mechanisms or relationships seen in real life. Assumptions made to assure that particular mechanisms are characterized can cause the model to be less valid with regard to other possible mechanisms. This makes the modeling of complex interrelationships more of an art than an exact science. For each particular decision problem, it is important to determine the assumptions to which each approach is sensitive, determine the appropriateness of these assumptions, and judge the relevance of the model sensitivity to them in the context of the decision problem and the forthcoming decisions that will result from it.

Conclusions

This study shows that the RISC model accurately predicts mortality and CVD events during the period of 5 years on which it is based (internal validity) and during an extended follow-up period up for 13 years (predictive validity). In addition, after recalibration, it accurately predicts mortality in the EPIC-Norfolk cohort as well (external validity). These findings provide a basis to generalize results from the RISC model.

Abbreviations

CABG: Coronary artery bypass graft; CHD: Coronary heart disease; CVD: Cardiovascular disease; EPIC: European Prospective Investigation of Cancer Study; ICD: International Classification of Diseases; PCI: Percutaneous coronary intervention; RISC: Rotterdam Ischemic Heart Disease and Stroke Computer Simulation.

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

MH coordinated and, with BK and BF, designed the study. BK and BF developed the simulation model and analyzed the data. AH, MB, EC, NW, and KK provided the data used in the model. BK and BF, MH, and ES contributed to the interpretation of the data. BK and BF wrote the manuscript. All authors read and approved the final manuscript

References

  1. Hayward RA, Krumholz HM, Zulman DM, Timbie JW, Vijan S: Optimizing statin treatment for primary prevention of coronary artery disease.

    Ann Intern Med 2010, 152:69-77. PubMed Abstract | Publisher Full Text OpenURL

  2. Pletcher MJ, Lazar L, Bibbins-Domingo K, Moran A, Rodondi N, Coxson P, Lightwood J, Williams L, Goldman L: Comparing impact and cost-effectiveness of primary prevention strategies for lipid-lowering.

    Ann Intern Med 2009, 150:243-254. PubMed Abstract | Publisher Full Text OpenURL

  3. Bibbins-Domingo K, Chertow GM, Coxson PG, Moran A, Lightwood JM, Pletcher MJ, Goldman L: Projected effect of dietary salt reductions on future cardiovascular disease.

    N Engl J Med 2010, 362:590-599. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  4. Hunink MGP, Siegel J, et al.: Decision making in health and medicine Integrating evidence and values. Cambridge: Cambridge University Press; 2001. OpenURL

  5. van Kempen BJH, Ferket BS, Spronk S, Hofman A, Steyerberg E, Hunink M: Do different methods of modeling statin treatment effectiveness influence the optimal decision?

    Med Decis Making 2012. OpenURL

  6. Philips Z, Ginnelly L, Sculpher M, Claxton K, Golder S, Riemsma R, Woolacoot N, Glanville J: Review of guidelines for good practice in decision-analytic modelling in health technology assessment.

    Health Technol Assess 2004, 8:iii-iv.

    ix-xi, 1-158

    PubMed Abstract | Publisher Full Text OpenURL

  7. Weinstein MC, O'Brien B, Hornberger J, Jackson J, Johannesson M, McCabe C, Luce BR, Studies ITFoGRP--M: Principles of good practice for decision analytic modeling in health-care evaluation: report of the ISPOR Task Force on Good Research Practices--modeling studies.

    Value Health 2003, 6:9-17. PubMed Abstract | Publisher Full Text OpenURL

  8. Goldhaber-Fiebert JD, Stout NK, Goldie SJ: Empirically evaluating decision-analytic models.

    Value Health 2010, 13:667-674. PubMed Abstract | Publisher Full Text OpenURL

  9. Eddy DM, Schlessinger L: Validation of the archimedes diabetes model.

    Diabetes Care 2003, 26:3102-3110. PubMed Abstract | Publisher Full Text OpenURL

  10. Palmer AJ, Roze S, Valentine WJ, Minshall ME, Foos V, Lurati FM, Lammert M, Spinas GA: Validation of the CORE Diabetes Model against epidemiological and clinical studies.

    Curr Med Res Opin 2004, 20(Suppl 1):S27-40. PubMed Abstract | Publisher Full Text OpenURL

  11. Kim LG, Thompson SG: Uncertainty and validation of health economic decision models.

    Health Econ 2010, 19:43-55. PubMed Abstract | Publisher Full Text OpenURL

  12. Welsing PM, Severens JL, Hartman M, van Gestel AM, van Riel PL, Laan RF: The initial validation of a Markov model for the economic evaluation of (new) treatments for rheumatoid arthritis.

    Pharmacoeconomics 2006, 24:1011-1020. PubMed Abstract | Publisher Full Text OpenURL

  13. Nijhuis RL, Stijnen T, Peeters A, Witteman JC, Hofman A, Hunink MG: Apparent and internal validity of a Monte Carlo-Markov model for cardiovascular disease in a cohort follow-up study.

    Med Decis Making 2006, 26:134-144. PubMed Abstract | Publisher Full Text OpenURL

  14. Hofman A, van Duijn CM, Franco OH, Ikram MA, Janssen HL, Klaver CC, Kuipers EJ, Nijsten TE, Stricker BH, Tiemeier H, et al.: The Rotterdam Study: 2012 objectives and design update.

    Eur J Epidemiol 2011, 26:657-686. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  15. Hofman A, Grobbee DE, de Jong PT, van den Ouweland FA: Determinants of disease and disability in the elderly: the Rotterdam Elderly Study.

    Eur J Epidemiol 1991, 7:403-422. PubMed Abstract | Publisher Full Text OpenURL

  16. Groot Koerkamp B, Weinstein MC, Stijnen T, Heijenbrok-Kal MH, Hunink MG: Uncertainty and patient heterogeneity in medical decision models.

    Med Decis Making 2010, 30:194-205. PubMed Abstract | Publisher Full Text OpenURL

  17. Groot Koerkamp B, Stijnen T, Weinstein MC, Hunink MG: The combined analysis of uncertainty and patient heterogeneity in medical decision models.

    Med Decis Making 2011, 31:650-661. PubMed Abstract | Publisher Full Text OpenURL

  18. Day N, Oakes S, Luben R, Khaw KT, Bingham S, Welch A, Wareham N: EPIC-Norfolk: study design and characteristics of the cohort. European Prospective Investigation of Cancer.

    Br J Cancer 1999, 80(Suppl 1):95-103. PubMed Abstract OpenURL

  19. Little R, An H: Robust likelihood-based analysis of multivariate data with missing values.

    Statistica Sinica 2004, 14:949-968. OpenURL

  20. Sinha S, Myint PK, Luben RN, Khaw KT: Accuracy of death certification and hospital record linkage for identification of incident stroke.

    BMC Med Res Methodol 2008, 8:74. PubMed Abstract | BioMed Central Full Text | PubMed Central Full Text OpenURL

  21. Hollander M, Koudstaal PJ, Bots ML, Grobbee DE, Hofman A, Breteler MM: Incidence, risk, and case fatality of first ever stroke in the elderly population. The Rotterdam Study.

    J Neurol Neurosurg Psychiatry 2003, 74:317-321. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  22. D'Agostino RB, Grundy S, Sullivan LM, Wilson P, Group CHDRP: Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation.

    JAMA 2001, 286:180-187. PubMed Abstract | Publisher Full Text OpenURL

  23. Unal B, Capewell S, Critchley JA: Coronary heart disease policy models: a systematic review.

    BMC Public Health 2006, 6:213. PubMed Abstract | BioMed Central Full Text | PubMed Central Full Text OpenURL

  24. Gunning-Schepers L: The health benefits of prevention: a simulation approach.

    Health Policy 1989, 12:1-255. PubMed Abstract OpenURL

  25. Hamilton VH, Racicot FE, Zowall H, Coupal L, Grover SA: The cost-effectiveness of HMG-CoA reductase inhibitors to prevent coronary heart disease. Estimating the benefits of increasing HDL-C.

    JAMA 1995, 273:1032-1038. PubMed Abstract | Publisher Full Text OpenURL

  26. Murray CJ, Lopez AD: Alternative projections of mortality and disability by cause 1990-2020: Global Burden of Disease Study.

    Lancet 1997, 349:1498-1504. PubMed Abstract | Publisher Full Text OpenURL

  27. Weinstein MC, Coxson PG, Williams LW, Pass TM, Stason WB, Goldman L: Forecasting coronary heart disease incidence, mortality, and cost: the Coronary Heart Disease Policy Model.

    Am J Public Health 1987, 77:1417-1426. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  28. Grover SA, Abrahamowicz M, Joseph L, Brewer C, Coupal L, Suissa S: The benefits of treating hyperlipidemia to prevent coronary heart disease. Estimating changes in life expectancy and morbidity.

    JAMA 1992, 267:816-822. PubMed Abstract | Publisher Full Text OpenURL

  29. Naidoo B, Thorogood M, McPherson K, Gunning-Schepers LJ: Modelling the effects of increased physical activity on coronary heart disease in England and Wales.

    J Epidemiol Community Health 1997, 51:144-150. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  30. Capewell S, Morrison CE, McMurray JJ: Contribution of modern cardiovascular treatment and risk factor changes to the decline in coronary heart disease mortality in Scotland between 1975 and 1994.

    Heart 1999, 81:380-386. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  31. Pencina MJ, D'Agostino RB, Larson MG, Massaro JM, Vasan RS: Predicting the 30-year risk of cardiovascular disease: the framingham heart study.

    Circulation 2009, 119:3078-3084. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  32. Ankle Brachial Index C, Fowkes FG, Murray GD, Butcher I, Heald CL, Lee RJ, Chambless LE, Folsom AR, Hirsch AT, Dramaix M, et al.: Ankle brachial index combined with Framingham Risk Score to predict cardiovascular events and mortality: a meta-analysis.

    JAMA 2008, 300:197-208. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  33. van Kempen BJ, Ferket BS, Hofman A, Spronk S, Steyerberg E, Hunink MG: Do different methods of modeling statin treatment effectiveness influence the optimal decision?

    Med Decis Making 2012, 32:507-516. PubMed Abstract | Publisher Full Text OpenURL

  34. Kopec JA, Fines P, Manuel DG, Buckeridge DL, Flanagan WM, Oderkirk J, Abrahamowicz M, Harper S, Sharif B, Okhmatovskaia A, et al.: Validation of population-based disease simulation models: a review of concepts and methods.

    BMC Public Health 2010, 10:710. PubMed Abstract | BioMed Central Full Text | PubMed Central Full Text OpenURL

Pre-publication history

The pre-publication history for this paper can be accessed here:

http://www.biomedcentral.com/1741-7015/10/158/prepub