Email updates

Keep up to date with the latest news and content from BMC Medical Research Methodology and BioMed Central.

Open Access Research article

Validating and updating a risk model for pneumonia – a case study

Ulrike Held1*, Daniel Sabanes Bové2, Johann Steurer1 and Leonhard Held2

Author Affiliations

1 Horten Centre for Patient Oriented Research and Knowledge Transfer, University Hospital Zurich, Zurich, Switzerland

2 Institute for Social and Preventive Medicine, Division of Biostatistics, University of Zurich, Zurich, Switzerland

For all author emails, please log on.

BMC Medical Research Methodology 2012, 12:99  doi:10.1186/1471-2288-12-99

Published: 20 July 2012

Abstract

Background

The development of risk prediction models is of increasing importance in medical research - their use in practice, however, is rare. Among other reasons this might be due to the fact that thorough validation is often lacking. This study focuses on two Bayesian approaches of how to validate a prediction rule for the diagnosis of pneumonia, and compares them with established validation methods.

Methods

Expert knowledge was used to derive a risk prediction model for pneumonia. Data on more than 600 patients presenting with cough and fever at a general practitioner’s practice in Switzerland were collected in order to validate the expert model and to examine the predictive performance of it. Additionally, four modifications of the original model including shrinkage of the regression coefficients, and two Bayesian approaches with the expert model used as prior mean and different weights for the prior covariance matrix were fitted. We quantify the predictive performance of the different methods with respect to calibration and discrimination, using cross-validation.

Results

The predictive performance of the unshrinked regression coefficients was poor when applied to the Swiss cohort. Shrinkage improved the results, but a Bayesian model formulation with unspecified weight of the informative prior lead to large AUC and small Brier score, naïve and after cross-validation. The advantage of this approach is the flexibility in case of a prior-data conflict.

Conclusions

Published risk prediction rules in clinical research need to be validated externally before they can be used in new settings. We propose to use a Bayesian model formulation with the original risk prediction rule as prior. The posterior means of the coefficients, given the validation data showed best predictive performance with respect to cross-validated calibration and discriminative ability.

Keywords:
Validation; Predictive performance; Bayesian model; g-factor; Pneumonia