Open Access Research article

Polytomous diagnosis of ovarian tumors as benign, borderline, primary invasive or metastatic: development and validation of standard and kernel-based risk prediction models

Ben Van Calster12, Lil Valentin3*, Caroline Van Holsbeke45, Antonia C Testa6, Tom Bourne47, Sabine Van Huffel1 and Dirk Timmerman4

Author Affiliations

1 Department of Electrical Engineering (ESAT-SISTA), Katholieke Universiteit Leuven, Kasteelpark Arenberg 10, B-3001 Leuven, Belgium

2 Leuvens Kankerinstituut, University Hospitals K.U. Leuven, Herestraat 49, B-3000 Leuven, Belgium

3 Department of Obstetrics and Gynecology, Skåne University Hospital Malmö, Lund University, SE20502 Malmö, Sweden

4 Department of Obstetrics and Gynecology, University Hospitals Leuven, Herestraat 49, B-3000 Leuven, Belgium

5 Department of Obstetrics and Gynecology, Ziekenhuis Oost-Limburg, Schiepse Bos 6, B-3600 Genk, Belgium

6 Istituto di Clinica Ostetrica e Gynecologica, Università Cattolica del Sacro Cuore, Largo Agostino Gemelli 8, Roma, Italy

7 Imperial College London, Hammersmith Campus, Du Cane Road, London W12 0HS, UK

For all author emails, please log on.

BMC Medical Research Methodology 2010, 10:96  doi:10.1186/1471-2288-10-96

Published: 20 October 2010



Hitherto, risk prediction models for preoperative ultrasound-based diagnosis of ovarian tumors were dichotomous (benign versus malignant). We develop and validate polytomous models (models that predict more than two events) to diagnose ovarian tumors as benign, borderline, primary invasive or metastatic invasive. The main focus is on how different types of models perform and compare.


A multi-center dataset containing 1066 women was used for model development and internal validation, whilst another multi-center dataset of 1938 women was used for temporal and external validation. Models were based on standard logistic regression and on penalized kernel-based algorithms (least squares support vector machines and kernel logistic regression). We used true polytomous models as well as combinations of dichotomous models based on the 'pairwise coupling' technique to produce polytomous risk estimates. Careful variable selection was performed, based largely on cross-validated c-index estimates. Model performance was assessed with the dichotomous c-index (i.e. the area under the ROC curve) and a polytomous extension, and with calibration graphs.


For all models, between 9 and 11 predictors were selected. Internal validation was successful with polytomous c-indexes between 0.64 and 0.69. For the best model dichotomous c-indexes were between 0.73 (primary invasive vs metastatic) and 0.96 (borderline vs metastatic). On temporal and external validation, overall discrimination performance was good with polytomous c-indexes between 0.57 and 0.64. However, discrimination between primary and metastatic invasive tumors decreased to near random levels. Standard logistic regression performed well in comparison with advanced algorithms, and combining dichotomous models performed well in comparison with true polytomous models. The best model was a combination of dichotomous logistic regression models. This model is available online.


We have developed models that successfully discriminate between benign, borderline, and invasive ovarian tumors. Methodologically, the combination of dichotomous models was an interesting approach to tackle the polytomous problem. Standard logistic regression models were not outperformed by regularized kernel-based alternatives, a finding to which the careful variable selection procedure will have contributed. The random discrimination between primary and metastatic invasive tumors on temporal/external validation demonstrated once more the necessity of validation studies.