Email updates

Keep up to date with the latest news and content from BMC Medical Research Methodology and BioMed Central.

Open Access Research article

Performance of statistical models to predict mental health and substance abuse cost

Maria Montez-Rath12*, Cindy L Christiansen23, Susan L Ettner4, Susan Loveland23 and Amy K Rosen23

Author affiliations

1 Boston University School of Public Health, Department of Biostatistics, Boston, Massachusetts, USA

2 Center for Health Quality, Outcomes and Economic Research, Bedford VAMC, Bedford, Massachusetts, USA

3 Boston University School of Public Health, Department of Health Services, Boston, Massachusetts, USA

4 University of California, Division of General Internal Medicine and Health Services Research, David Geffen School of Medicine, Los Angeles, California, USA

For all author emails, please log on.

Citation and License

BMC Medical Research Methodology 2006, 6:53  doi:10.1186/1471-2288-6-53

Published: 26 October 2006

Abstract

Background

Providers use risk-adjustment systems to help manage healthcare costs. Typically, ordinary least squares (OLS) models on either untransformed or log-transformed cost are used. We examine the predictive ability of several statistical models, demonstrate how model choice depends on the goal for the predictive model, and examine whether building models on samples of the data affects model choice.

Methods

Our sample consisted of 525,620 Veterans Health Administration patients with mental health (MH) or substance abuse (SA) diagnoses who incurred costs during fiscal year 1999. We tested two models on a transformation of cost: a Log Normal model and a Square-root Normal model, and three generalized linear models on untransformed cost, defined by distributional assumption and link function: Normal with identity link (OLS); Gamma with log link; and Gamma with square-root link. Risk-adjusters included age, sex, and 12 MH/SA categories. To determine the best model among the entire dataset, predictive ability was evaluated using root mean square error (RMSE), mean absolute prediction error (MAPE), and predictive ratios of predicted to observed cost (PR) among deciles of predicted cost, by comparing point estimates and 95% bias-corrected bootstrap confidence intervals. To study the effect of analyzing a random sample of the population on model choice, we re-computed these statistics using random samples beginning with 5,000 patients and ending with the entire sample.

Results

The Square-root Normal model had the lowest estimates of the RMSE and MAPE, with bootstrap confidence intervals that were always lower than those for the other models. The Gamma with square-root link was best as measured by the PRs. The choice of best model could vary if smaller samples were used and the Gamma with square-root link model had convergence problems with small samples.

Conclusion

Models with square-root transformation or link fit the data best. This function (whether used as transformation or as a link) seems to help deal with the high comorbidity of this population by introducing a form of interaction. The Gamma distribution helps with the long tail of the distribution. However, the Normal distribution is suitable if the correct transformation of the outcome is used.