Polychotomization of continuous variables in regression models based on the overall <it>C </it>index

1472-6947-6-41 1472-6947 Research article Polychotomization of continuous variables in regression models based on the overall <it>C </it>index Tsuruta Harukazu ts@med.kitasato-u.ac.jp Bax Leon leonbax@kitasato-u.ac.jp

Department of Medical Informatics, School of Allied Health Sciences, Kitasato University, Sagamihara, Kanagawa, 228-8555, Japan

Department of Medical Informatics, Graduate School of Medical Sciences, Kitasato University, Sagamihara, Kanagawa, 228-8555, Japan

BMC Medical Informatics and Decision Making 1472-6947 2006 6 1 41 http://www.biomedcentral.com/1472-6947/6/41 17169154 10.1186/1472-6947-6-41 23 5 2006 14 12 2006 14 12 2006 2006 Tsuruta and Bax; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Background

When developing multivariable regression models for diagnosis or prognosis, continuous independent variables can be categorized to make a prediction table instead of a prediction formula. Although many methods have been proposed to dichotomize prognostic variables, to date there has been no integrated method for polychotomization. The latter is necessary when dichotomization results in too much loss of information or when central values refer to normal states and more dispersed values refer to less preferable states, a situation that is not unusual in medical settings (e.g. body temperature, blood pressure). The goal of our study was to develop a theoretical and practical method for polychotomization.

Methods

We used the overall discrimination index C, introduced by Harrel, as a measure of the predictive ability of an independent regressor variable and derived a method for polychotomization mathematically. Since the naïve application of our method, like some existing methods, gives rise to positive bias, we developed a parametric method that minimizes this bias and assessed its performance by the use of Monte Carlo simulation.

Results

The overall C is closely related to the area under the ROC curve and the produced di(poly)chotomized variable's predictive performance is comparable to the original continuous variable. The simulation shows that the parametric method is essentially unbiased for both the estimates of performance and the cutoff points. Application of our method to the predictor variables of a previous study on rhabdomyolysis shows that it can be used to make probability profile tables that are applicable to the diagnosis or prognosis of individual patient status.

Conclusion

We propose a polychotomization (including dichotomization) method for independent continuous variables in regression models based on the overall discrimination index C and clarified its meaning mathematically. To avoid positive bias in application, we have proposed and evaluated a parametric method. The proposed method for polychotomizing continuous regressor variables performed well and can be used to create probability profile tables.

Background

In modern diagnostic and descriptive prognostic research, regression models are often used to model an illness-related outcome based on a number of independent regressor variables, also referred to as diagnostic indicators or prognostic predictors 1. Such regressor variables can be categorical or numerical. From the vantage point of applicability in a clinical setting, categorization (often dichotomization) of continuous independent variables can be useful. Obtaining a prediction at the bedside without computer is easier with a prediction table based on categorized variables than with a prediction formula. Even if calculation is not problematic, table presentation of the risks has the practical advantages that (1) repeated use of the table will give physicians an intuitive feel for the disease risk, and (2) even if the value of one or two of the prognostic variables is not available, physicians can obtain a probability range corresponding to the patient's risk by referring to the most extreme cases in the table.

Depending on the setting, several different approaches have been proposed for dichotomization. One popular method is to find a cutoff point to discriminate whether a patient belongs to a normal group or a disease group based on the observed value of a predictive factor. This type of discriminant function analysis was first developed by R.A. Fisher 2 in 1930's. The Mahalanobis distance 3 can be used to find the optimal cutoff point if the variable distributes normally.

Another solution, sometimes used in clinical chemistry, is to find a cutoff point that maximizes the sum of sensitivity (SE) and specificity (SP) 45. There are different versions of this approach where one can maximize the weighted sum of SE and SP, or maximize the SE while fixing SP to an acceptable value 67. Cantor claimed that these methods have been used in many published articles without giving a theoretical foundation or scientific justification 8.

Yet another straightforward and popular method is to select a classification that maximizes a measure of difference between the two groups, such as the p-value of a chi square statistic 910. This method, sometimes called the minimum p-value approach, has been described and used for the prognosis of cancers 1112. Several authors have pointed out that the naïve selection used in this method overestimates the significance of the predictor or indicator's relationship to the dependent variable because of multiple testing, and several adjustment methods of the observed p-values have been proposed 9101112131415161718.

Besides using the data at hand to come to a dichotomization of continuous variables, it is also possible to use profit (benefit) or loss (cost) information. In that case, the optical cutoff point is defined so as to maximize the expected utility. Metz showed that the optimal point is the spot on the ROC curve at which the slope is (L/B)(1-p)/p, where B is the net benefit of treating diseased individuals, L the net loss of treating non-diseased individuals, and p the prevalence of the disease under study 19. Nevertheless, Cantor et al., in a review of studies in the medical literature that referred to "ROC" and "cutoff", found that only a few articles included a L/B ratio in the analysis for determining an optimal cutoff point 8.

The above methods all concern dichotomization. However, when central values refer to normal states and dispersed values to diseased states, two (or more) cutoff points are necessary to discriminate these states. Consequently, one is inevitably faced with the challenge of polychotomization. Unfortunately, methods for polychotomization are less developed. Although Kristjansson et al. 20 described a method for choosing optimal cutoff points in a screening test with a continuous score to divide people into a number of disease categories, their method is not applicable to polychotomization of regressor variables in regression models; their criterion loses its meaning in this setting.

The major goal of our study is to develop a theoretical and practical method for polychotomization. We propose a novel approach for independent continuous variables in regression models based on the overall discrimination index C introduced by Harrel et al. 2122. We will show that this index is closely related to the area under the ROC curve for the original continuous variable and that the resulting categorized variables have predictive properties comparable to the original continuous variable. However, the naïve search of the maximum C index gives rise to positive bias, not unlike the minimum p-value approach 9101112131415161718 or the method of maximizing the sum of the sensitivity and specificity 45. We therefore propose a parametric version in which the estimates of the predictive performance and cutoff points are both essentially unbiased. We evaluate this method and present means and standard deviations of predictive performance and cutoff point estimates for typical cases via Monte Carlo simulation. Finally, we provide a simple application example with a predictive regression model for rhabdomyolysis and show how our method can be used to create a probability profile table.

Methods

The categorization criterion

We assume there is an existing predictive model based on patients that belong to either a normal group or a diseased group and that the distribution of the relevant independent continuous variable X is known or that we have observations on it. Our goal is to find a method of optimal polychotomization for this continuous variable with a minimum loss of predictive ability. This involves making the number of possible patient's profiles finite, and replacing the regression formula with a table of the risk probabilities for all patient profiles. Different from most previously developed approaches we have no a priori intention to categorize the variable into two classes and we assume that it might be necessary to compare categorizations to three or more classes.

For this discussion we need a measure to evaluate the predictive power of a predictive variable. Our choice for a measure of predictive power is the overall discrimination index C 21222324, or the 'pair consistency probability', as we like to call it. This measure refers to the probability that the relative position of single normal-disease pair values is consistent with the relative position of their values of central tendency.

Without losing generality, we assume that the central value of the distribution of the random variable X in the group of healthy cases is smaller than the central value in the group of diseased cases. Next we take a sample x_i[h]from the healthy group and another sample x_i[d]from the diseased group randomly. Then the pair (x_i[h], x_i[d]) is considered consistent if x_i[h]<x_i[d], tied if x_i[h]= x_i[d], and inconsistent if x_i[h]> x_i[d]and the pair consistency probability C is defined as:

C = p c o n + 1 2 p t i e d , ( 1 ) MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqWGdbWqcqGH9aqpcqWGWbaCdaWgaaWcbaGaem4yamMaem4Ba8MaemOBa4gabeaakiabgUcaRmaalaaabaGaeGymaedabaGaeGOmaidaaiabdchaWnaaBaaaleaacqWG0baDcqWGPbqAcqWGLbqzcqWGKbazaeqaaOGaeiilaWIaaCzcaiaaxMaadaqadaqaaiabigdaXaGaayjkaiaawMcaaaaa@42FB@

where p_conand p_tieddenote the probabilities that the pair is consistent and tied respectively.

Next, if we let f_hrepresent the probability density function (PDF) of X in the healthy group and f_drepresent the PDF of X in the diseased group, and let z represent a cutoff point for dichotomization, then the true positive fraction Tp and false positive fraction Fp are defined by

T p = ∫ z ∞ f d ( x ) d x and F p = ∫ z ∞ f h ( x ) d x . MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqWGubavcqWGWbaCcqGH9aqpdaWdXaqaaiabdAgaMnaaBaaaleaacqWGKbazaeqaaOGaeiikaGIaemiEaGNaeiykaKcaleaacqWG6bGEaeaacqGHEisPa0Gaey4kIipakiabdsgaKjabdIha4jabbccaGiabbggaHjabb6gaUjabbsgaKjabbccaGiabdAeagjabdchaWjabg2da9maapedabaGaemOzay2aaSbaaSqaaiabdIgaObqabaGccqGGOaakcqWG4baEcqGGPaqkaSqaaiabdQha6bqaaiabg6HiLcqdcqGHRiI8aOGaemizaqMaemiEaGNaeiOla4caaa@5629@

In the case that the variable is continuous, as z increases, Tp and Fp both decrease continuously. The ROC curve 1925 can be depicted as the trace of points (Fp , Tp ). Green and Swets 25 demonstrated that

C = ∫ − ∞ ∞ P ( x [ h ] = z ) ⋅ P ( x [ d ] > z ) d z = ∫ − ∞ ∞ f h ( z ) ⋅ [ ∫ z ∞ f d ( x ) d x ] d z = ∫ 0 1 T p ( z ) d F p ( z ) . MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakqaaeeqaaiabdoeadjabg2da9maapedabaGaemiuaaLaeiikaGIaemiEaG3aaSbaaSqaaiabcUfaBjabdIgaOjabc2faDbqabaGccqGH9aqpcqWG6bGEcqGGPaqkcqGHflY1cqWGqbaucqGGOaakcqWG4baEdaWgaaWcbaGaei4waSLaemizaqMaeiyxa0fabeaakiabg6da+iabdQha6jabcMcaPaWcbaGaeyOeI0IaeyOhIukabaGaeyOhIukaniabgUIiYdGccqWGKbazcqWG6bGEaeaacqGH9aqpdaWdXaqaaiabdAgaMnaaBaaaleaacqWGObaAaeqaaOGaeiikaGIaemOEaONaeiykaKIaeyyXICTaei4waS1aa8qmaeaacqWGMbGzdaWgaaWcbaGaemizaqgabeaakiabcIcaOiabdIha4jabcMcaPiabdsgaKjabdIha4jabc2faDjabdsgaKjabdQha6bWcbaGaemOEaOhabaGaeyOhIukaniabgUIiYdaaleaacqGHsislcqGHEisPaeaacqGHEisPa0Gaey4kIipaaOqaaiabg2da9maapedabaGaemivaqLaemiCaaNaeiikaGIaemOEaONaeiykaKIaemizaqMaemOrayKaemiCaaNaeiikaGIaemOEaONaeiykaKIaeiOla4caleaacqaIWaamaeaacqaIXaqma0Gaey4kIipaaaaa@84FF@

This means that the pair consistency probability is equivalent to the area under the ROC curve for continuous variables. We will demonstrate that this relation also holds for polychotomized variables, and that the pair consistency probability C is a good measure to compare the predictive ability with the original continuous variable.

Optimal cutoff point for dichotomization

First, we discuss our method for dichotomization in which a continuous independent variable in a predictive model is categorized to one of two classes by a cutoff point. If we denote the value of the cutoff point z and assume that X is continuous in both the healthy and the diseased groups, that is, P(x_[h]= z) = 0 and P(x_[d]= z) = 0, the results of random pair sampling are classified into the following four cases:

x_[h]<z and x_[d]<z, tied

x_[h]<z and x_[d]> z, consistent

x_[h]> z and x_[d]<z, inconsistent

x_[h]> z and x_[d]> z, tied.

Let α denote the probability that x_[h]is greater than z, and β denote the probability that x_[d]is less than z. Assuming that the central value of the distribution of the random variable X in the group of healthy cases is smaller than the central value in the group of diseased cases, we have

α = ∫ z ∞ f h ( x ) d x = F p and β = ∫ − ∞ z f d ( x ) d x = 1 − T p . ( 2 ) MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaaiiGacqWFXoqycqGH9aqpdaWdXaqaaiabdAgaMnaaBaaaleaacqWGObaAaeqaaOGaeiikaGIaemiEaGNaeiykaKIaemizaqMaemiEaGNaeyypa0JaemOrayKaemiCaaNaeeiiaaIaeeyyaeMaeeOBa4MaeeizaqMaeeiiaaIae8NSdiMaeyypa0Zaa8qmaeaacqWGMbGzdaWgaaWcbaGaemizaqgabeaakiabcIcaOiabdIha4jabcMcaPiabdsgaKjabdIha4jabg2da9iabigdaXiabgkHiTiabdsfaujabdchaWjabc6caUiaaxMaacaWLjaWaaeWaaeaacqaIYaGmaiaawIcacaGLPaaaaSqaaiabgkHiTiabg6HiLcqaaiabdQha6bqdcqGHRiI8aaWcbaGaemOEaOhabaGaeyOhIukaniabgUIiYdaaaa@61EC@

Then the probability of a consistent pair becomes

p_con= (1 - α)(1 - β),

and the probability of a tied pair becomes

p_tied= (1 - α) β + α (1 - β).

Assigning these probabilities into (1), we have

C = 1 - (α + β)/2. (3)

It follows that the highest pair consistency probability is achieved when the sum of the two types of errors, α + β, is minimized. Since sensitivity is (1 - β) and specificity is (1 - α), we have

C = (sensitivity + specificity)/2. (4)

Therefore the highest pair consistency probability is achieved when the sum of sensitivity and specificity is maximized.

Figure 1 illustrates the changes of C when f_hand f_dare normal. Let z be the cutoff point where f_hand f_dcross between two peaks. If the cutoff point is shifted to the right from z, then α will decrease and β will increase. In this case, since f_dis greater than f_hin this interval, the increase of β is greater than the decrease of α. If the cutoff point is shifted to the left, then the opposite is true. Therefore, the sum of the two types of errors, α + β, occupies the local minimum at the point where f_hand f_dintersect between the peaks. If f_hand f_dare unimodal and cross only at one point, α + β occupies the true minimum at the cross point.

Figure 1

Sample illustration of the change of pair consistency probability C

Sample illustration of the change of pair consistency probability C. Lower curves: sample illustration of the probability density functions in the healthy group (f_h) and in the diseased group (f_d); Upper curve: pair consistency probability C (=(1- (α + β)/2)) as a function of cutoff point z. The sum of the two types of errors, α + β, takes a local minimum at the point where f_hand f_dintersect.

Generation and meaning of the ROC straight line graph for a dichotomous variable

As we have described earlier, when the independent variable is continuous, Tp and Fp both decrease continuously and the ROC curve can be depicted as the trace of points (Fp , Tp ). But what happens to the ROC curve when the variable is dichotomous? Let z₀represent the cutoff point and Fp₀and Tp₀denote the false positive and true positive fractions for z₀, respectively. Unlike the continuous variables, only three points (1, 1), (Fp₀, Tp₀) and (0, 0) are depicted in Fp - Tp coordinates and we cannot obtain a true curve (see Figure 2). We jointed these points with straight lines, and labelled this graph the ROC straight line graph. Then area A under the ROC straight line graph becomes:

Figure 2

The ROC curve and ROC straight line graph for the sample distributions in Figure 1

The ROC curve and ROC straight line graph for the sample distributions in Figure 1. The ROC curve was derived from the distributions in Figure 1 and a ROC straight line graph for the cutoff point z₀, which gives the maximum C, was also plotted. Filled part A shows the area under the ROC straight line graph.

A = Fp₀Tp₀/2 + (1 - Fp₀)Tp₀+ (1 - Fp₀)(1 - Tp₀)/2

= 1 - (α + β)/2 = C. (5)

This means that for a dichotomous variable, the area under the ROC straight line graph for a dichotomous variable is, analogous to the case with a continuous variable, equivalent to the pair consistency probability C. Therefore, finding a cutoff point that maximizes C is equivalent to the problem of finding the point (Fp₀, Tp₀) on the original ROC curve that maximizes the area A under the ROC straight line graph.

Optimal cutoff points for polychotomization

Next, consider the polychotomous case. Again, let x_[h]be a sample from the continuous random variable X in the healthy group and x_[d]a sample from the same variable in the disease group, both taken randomly. Let z₀= -∞, z_n= ∞ and z₁, z₂,..., z_n-1be cutoff points where z₁<z₂<...<z_n-1. We define that

H k = P ( z k − 1 < x [ h ] < z k ) = ∫ z k − 1 z k f h ( x ) d x ( k = 1 , ... , n ) MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaafaqabeqacaaabaGaemisaG0aaSbaaSqaaiabdUgaRbqabaGccqGH9aqpcqWGqbaucqGGOaakcqWG6bGEdaWgaaWcbaGaem4AaSMaeyOeI0IaeGymaedabeaakiabgYda8iabdIha4naaBaaaleaacqGGBbWwcqWGObaAcqGGDbqxaeqaaOGaeyipaWJaemOEaO3aaSbaaSqaaiabdUgaRbqabaGccqGGPaqkcqGH9aqpdaWdXaqaaiabdAgaMnaaBaaaleaacqWGObaAaeqaaOGaeiikaGIaemiEaGNaeiykaKcaleaacqWG6bGEdaWgaaadbaGaem4AaSMaeyOeI0IaeGymaedabeaaaSqaaiabdQha6naaBaaameaacqWGRbWAaeqaaaqdcqGHRiI8aOGaemizaqMaemiEaGhabaGaeiikaGIaem4AaSMaeyypa0JaeGymaeJaeiilaWIaeiOla4IaeiOla4IaeiOla4IaeiilaWIaemOBa4MaeiykaKcaaaaa@61BD@

D k = P ( z k − 1 < x [ d ] < z k ) = ∫ z k − 1 z k f d ( x ) d x ( k = 1 , ... , n ) . MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaafaqabeqacaaabaGaemiraq0aaSbaaSqaaiabdUgaRbqabaGccqGH9aqpcqWGqbaucqGGOaakcqWG6bGEdaWgaaWcbaGaem4AaSMaeyOeI0IaeGymaedabeaakiabgYda8iabdIha4naaBaaaleaacqGGBbWwcqWGKbazcqGGDbqxaeqaaOGaeyipaWJaemOEaO3aaSbaaSqaaiabdUgaRbqabaGccqGGPaqkcqGH9aqpdaWdXaqaaiabdAgaMnaaBaaaleaacqWGKbazaeqaaOGaeiikaGIaemiEaGNaeiykaKcaleaacqWG6bGEdaWgaaadbaGaem4AaSMaeyOeI0IaeGymaedabeaaaSqaaiabdQha6naaBaaameaacqWGRbWAaeqaaaqdcqGHRiI8aOGaemizaqMaemiEaGhabaGaeiikaGIaem4AaSMaeyypa0JaeGymaeJaeiilaWIaeiOla4IaeiOla4IaeiOla4IaeiilaWIaemOBa4MaeiykaKIaeiOla4caaaaa@6289@

Then the probabilities for tied and concordant pairs become

p t i e d = ∑ k = 1 n H k ⋅ D k and p c o n = ∑ k = 1 n − 1 H k ⋅ ( ∑ j = k + 1 n D j ) , MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqWGWbaCdaWgaaWcbaGaemiDaqNaemyAaKMaemyzauMaemizaqgabeaakiabg2da9maaqahabaGaemisaG0aaSbaaSqaaiabdUgaRbqabaGccqGHflY1cqWGebardaWgaaWcbaGaem4AaSgabeaaaeaacqWGRbWAcqGH9aqpcqaIXaqmaeaacqWGUbGBa0GaeyyeIuoakiabbccaGiabbggaHjabb6gaUjabbsgaKjabbccaGiabdchaWnaaBaaaleaacqWGJbWycqWGVbWBcqWGUbGBaeqaaOGaeyypa0ZaaabCaeaacqWGibasdaWgaaWcbaGaem4AaSgabeaakiabgwSixpaabmaabaWaaabCaeaacqWGebardaWgaaWcbaGaemOAaOgabeaaaeaacqWGQbGAcqGH9aqpcqWGRbWAcqGHRaWkcqaIXaqmaeaacqWGUbGBa0GaeyyeIuoaaOGaayjkaiaawMcaaaWcbaGaem4AaSMaeyypa0JaeGymaedabaGaemOBa4MaeyOeI0IaeGymaedaniabggHiLdGccqGGSaalaaa@6BB1@

and the pair consistency probability C can be calculated from equation (1).

We also define

Tp_k= P (x_[d]> z_k) and Fp_k= P (x_[h]> z_k) (k = 0,..., n).

The points (Fp_k, Tp_k) lie on the original ROC curve, and the set of points (Fp_k, Tp_k) jointed by straight lines yields the ROC straight line graph. Let A represent the area under the ROC straight line graph and A_krepresent the area under the line whose ends are (Fp_k-1, Tp_k-1) and (Fp_k, Tp_k). As illustrated in Figure 3, the area A_kis

Figure 3

Area A_kunder the ROC straight line graph

Area A_kunder the ROC straight line graph. The filled part shows the area A_kunder the ROC straight line graph with end points (Fp_k-1, Tp_k-1) and (Fp_k, Tp_k).

A k = 1 2 { F p k − 1 − F p k } ⋅ { T p k − 1 + T p k } . MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqWGbbqqdaWgaaWcbaGaem4AaSgabeaakiabg2da9maalaaabaGaeGymaedabaGaeGOmaidaaiabcUha7jabdAeagjabdchaWnaaBaaaleaacqWGRbWAcqGHsislcqaIXaqmaeqaaOGaeyOeI0IaemOrayKaemiCaa3aaSbaaSqaaiabdUgaRbqabaGccqGG9bqFcqGHflY1cqGG7bWEcqWGubavcqWGWbaCdaWgaaWcbaGaem4AaSMaeyOeI0IaeGymaedabeaakiabgUcaRiabdsfaujabdchaWnaaBaaaleaacqWGRbWAaeqaaOGaeiyFa0NaeiOla4caaa@517F@

Therefore,

A k = 1 2 P ( z k − 1 < x [ h ] < z k ) ⋅ { P ( x [ d ] > z k − 1 ) + P ( x [ d ] > z k ) } = P ( z k − 1 < x [ h ] < z k ) ⋅ P ( x [ d ] > z k ) + 1 2 P ( z k − 1 < x [ h ] < z k ) ⋅ P ( z k − 1 < x [ d ] < z k ) = { H k ⋅ ( ∑ j = k + 1 n D j ) + 1 2 H k ⋅ D k ( k = 1 , ... , n − 1 ) 1 2 H n ⋅ D n ( k = n ) MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakqaabeqaaiabdgeabnaaBaaaleaacqWGRbWAaeqaaOGaeyypa0ZaaSaaaeaacqaIXaqmaeaacqaIYaGmaaGaemiuaaLaeiikaGIaemOEaO3aaSbaaSqaaiabdUgaRjabgkHiTiabigdaXaqabaGccqGH8aapcqWG4baEdaWgaaWcbaGaei4waSLaemiAaGMaeiyxa0fabeaakiabgYda8iabdQha6naaBaaaleaacqWGRbWAaeqaaOGaeiykaKIaeyyXICTaei4EaSNaemiuaaLaeiikaGIaemiEaG3aaSbaaSqaaiabcUfaBjabdsgaKjabc2faDbqabaGccqGH+aGpcqWG6bGEdaWgaaWcbaGaem4AaSMaeyOeI0IaeGymaedabeaakiabcMcaPiabgUcaRiabdcfaqjabcIcaOiabdIha4naaBaaaleaacqGGBbWwcqWGKbazcqGGDbqxaeqaaOGaeyOpa4JaemOEaO3aaSbaaSqaaiabdUgaRbqabaGccqGGPaqkcqGG9bqFaeaacqGH9aqpcqWGqbaucqGGOaakcqWG6bGEdaWgaaWcbaGaem4AaSMaeyOeI0IaeGymaedabeaakiabgYda8iabdIha4naaBaaaleaacqGGBbWwcqWGObaAcqGGDbqxaeqaaOGaeyipaWJaemOEaO3aaSbaaSqaaiabdUgaRbqabaGccqGGPaqkcqGHflY1cqWGqbaucqGGOaakcqWG4baEdaWgaaWcbaGaei4waSLaemizaqMaeiyxa0fabeaakiabg6da+iabdQha6naaBaaaleaacqWGRbWAaeqaaOGaeiykaKIaey4kaSYaaSaaaeaacqaIXaqmaeaacqaIYaGmaaGaemiuaaLaeiikaGIaemOEaO3aaSbaaSqaaiabdUgaRjabgkHiTiabigdaXaqabaGccqGH8aapcqWG4baEdaWgaaWcbaGaei4waSLaemiAaGMaeiyxa0fabeaakiabgYda8iabdQha6naaBaaaleaacqWGRbWAaeqaaOGaeiykaKIaeyyXICTaemiuaaLaeiikaGIaemOEaO3aaSbaaSqaaiabdUgaRjabgkHiTiabigdaXaqabaGccqGH8aapcqWG4baEdaWgaaWcbaGaei4waSLaemizaqMaeiyxa0fabeaakiabgYda8iabdQha6naaBaaaleaacqWGRbWAaeqaaOGaeiykaKcabaGaeyypa0ZaaiqabeaafaqaaeGacaaabaGaemisaG0aaSbaaSqaaiabdUgaRbqabaGccqGHflY1daqadaqaamaaqahabaGaemiraq0aaSbaaSqaaiabdQgaQbqabaaabaGaemOAaOMaeyypa0Jaem4AaSMaey4kaSIaeGymaedabaGaemOBa4ganiabggHiLdaakiaawIcacaGLPaaacqGHRaWkdaWcaaqaaiabigdaXaqaaiabikdaYaaacqWGibasdaWgaaWcbaGaem4AaSgabeaakiabgwSixlabdseaenaaBaaaleaacqWGRbWAaeqaaaGcbaGaeiikaGIaem4AaSMaeyypa0JaeGymaeJaeiilaWIaeiOla4IaeiOla4IaeiOla4IaeiilaWIaemOBa4MaeyOeI0IaeGymaeJaeiykaKcabaWaaSaaaeaacqaIXaqmaeaacqaIYaGmaaGaemisaG0aaSbaaSqaaiabd6gaUbqabaGccqGHflY1cqWGebardaWgaaWcbaGaemOBa4gabeaaaOqaaiabcIcaOiabdUgaRjabg2da9iabd6gaUjabcMcaPaaaaiaawUhaaaaaaa@EC8E@

Then we have

A = ∑ k = 1 n A k = ∑ k = 1 n − 1 H k ⋅ ( ∑ j = k + 1 n D j ) + 1 2 ∑ k = 1 n H k ⋅ D k = p c o n + 1 2 p t i e d = C . ( 6 ) MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaafaqadeWabaaabaGaemyqaeKaeyypa0ZaaabCaeaacqWGbbqqdaWgaaWcbaGaem4AaSgabeaaaeaacqWGRbWAcqGH9aqpcqaIXaqmaeaacqWGUbGBa0GaeyyeIuoaaOqaaiabg2da9maaqahabaGaemisaG0aaSbaaSqaaiabdUgaRbqabaGccqGHflY1daqadaqaamaaqahabaGaemiraq0aaSbaaSqaaiabdQgaQbqabaaabaGaemOAaOMaeyypa0Jaem4AaSMaey4kaSIaeGymaedabaGaemOBa4ganiabggHiLdaakiaawIcacaGLPaaacqGHRaWkdaWcaaqaaiabigdaXaqaaiabikdaYaaadaaeWbqaaiabdIeainaaBaaaleaacqWGRbWAaeqaaOGaeyyXICTaemiraq0aaSbaaSqaaiabdUgaRbqabaaabaGaem4AaSMaeyypa0JaeGymaedabaGaemOBa4ganiabggHiLdaaleaacqWGRbWAcqGH9aqpcqaIXaqmaeaacqWGUbGBcqGHsislcqaIXaqma0GaeyyeIuoaaOqaaiabg2da9iabdchaWnaaBaaaleaacqWGJbWycqWGVbWBcqWGUbGBaeqaaOGaey4kaSYaaSaaaeaacqaIXaqmaeaacqaIYaGmaaGaemiCaa3aaSbaaSqaaiabdsha0jabdMgaPjabdwgaLjabdsgaKbqabaGccqGH9aqpcqWGdbWqcqGGUaGlaaGaaCzcaiaaxMaadaqadaqaaiabiAda2aGaayjkaiaawMcaaaaa@7D59@

Again, the pair consistency probability C for the polychotomized variable is equivalent to the area under its ROC straight line graph, and the problem of finding the optimal cutoff points that maximize C is mathematically equivalent to finding the set of edge points of the ROC straight line graph that maximizes the area A under that graph.

Optimal cutoff points for variables for which normal and diseased cases have a common central tendency

There are many predictive variables whose central values refer to a normal state and whose more dispersed values refer to less preferable states. In the example of rhabdomyolysis prognosis that will follow later, body temperature, pulse rate, plasma sodium, and plasma pH are such variables. For these predictors, we need to find at least two cutoff points to discriminate normal and abnormal states. If we denote the values of the cutoff points z₁and z₂(z₁<z₂), and regard the value between these two cutoff points as normal, then type I error α and type II error β become:

α = ∫ − ∞ z 1 f h ( x ) d x + ∫ z 2 ∞ f h ( x ) d x = F p MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaaiiGacqWFXoqycqGH9aqpdaWdXaqaaiabdAgaMnaaBaaaleaacqWGObaAaeqaaOGaeiikaGIaemiEaGNaeiykaKcaleaacqGHsislcqGHEisPaeaacqWG6bGEdaWgaaadbaGaeGymaedabeaaa0Gaey4kIipakiabdsgaKjabdIha4jabgUcaRmaapedabaGaemOzay2aaSbaaSqaaiabdIgaObqabaGccqGGOaakcqWG4baEcqGGPaqkaSqaaiabdQha6naaBaaameaacqaIYaGmaeqaaaWcbaGaeyOhIukaniabgUIiYdGccqWGKbazcqWG4baEcqGH9aqpcqWGgbGrcqWGWbaCaaa@52E6@

and

β = ∫ z 1 z 2 f d ( x ) d x = 1 − T p . MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaaiiGacqWFYoGycqGH9aqpdaWdXaqaaiabdAgaMnaaBaaaleaacqWGKbazaeqaaOGaeiikaGIaemiEaGNaeiykaKcaleaacqWG6bGEdaWgaaadbaGaeGymaedabeaaaSqaaiabdQha6naaBaaameaacqaIYaGmaeqaaaqdcqGHRiI8aOGaemizaqMaemiEaGNaeyypa0JaeGymaeJaeyOeI0IaemivaqLaemiCaaNaeiOla4caaa@4600@

The pair consistency probability C can now be calculated with equation (3) and the combination of cutoff points (z₁, z₂) which maximizes (3) becomes the solution. In case of categorization of the variable into more than three states, we can define the optimal combination of cutoff points as follows: Let z_n= -∞, w_n= ∞ and z₁, z₂,..., z_n-1, w₁, w₂,..., w_n-1be cutoff points where z_n-1<...<z₂<z₁<w₁<w₂<...<w_n-1, and

H 1 = ∫ z 1 w 1 f h ( x ) d x , D 1 = ∫ z 1 w 1 f d ( x ) d x MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaafaqabeqacaaabaGaemisaG0aaSbaaSqaaiabigdaXaqabaGccqGH9aqpdaWdXaqaaiabdAgaMnaaBaaaleaacqWGObaAaeqaaOGaeiikaGIaemiEaGNaeiykaKIaemizaqMaemiEaGNaeiilaWcaleaacqWG6bGEdaWgaaadbaGaeGymaedabeaaaSqaaiabdEha3naaBaaameaacqaIXaqmaeqaaaqdcqGHRiI8aaGcbaGaemiraq0aaSbaaSqaaiabigdaXaqabaGccqGH9aqpdaWdXaqaaiabdAgaMnaaBaaaleaacqWGKbazaeqaaOGaeiikaGIaemiEaGNaeiykaKIaemizaqMaemiEaGhaleaacqWG6bGEdaWgaaadbaGaeGymaedabeaaaSqaaiabdEha3naaBaaameaacqaIXaqmaeqaaaqdcqGHRiI8aaaaaaa@5493@

H k = ∫ z k z k − 1 f h ( x ) d x + ∫ w k − 1 w k f h ( x ) d x ( k = 2 , ... , n ) , MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaafaqabeqacaaabaGaemisaG0aaSbaaSqaaiabdUgaRbqabaGccqGH9aqpdaWdXaqaaiabdAgaMnaaBaaaleaacqWGObaAaeqaaOGaeiikaGIaemiEaGNaeiykaKIaemizaqMaemiEaGNaey4kaSYaa8qmaeaacqWGMbGzdaWgaaWcbaGaemiAaGgabeaakiabcIcaOiabdIha4jabcMcaPiabdsgaKjabdIha4bWcbaGaem4DaC3aaSbaaWqaaiabdUgaRjabgkHiTiabigdaXaqabaaaleaacqWG3bWDdaWgaaadbaGaem4AaSgabeaaa0Gaey4kIipaaSqaaiabdQha6naaBaaameaacqWGRbWAaeqaaaWcbaGaemOEaO3aaSbaaWqaaiabdUgaRjabgkHiTiabigdaXaqabaaaniabgUIiYdaakeaacqGGOaakcqWGRbWAcqGH9aqpcqaIYaGmcqGGSaalcqGGUaGlcqGGUaGlcqGGUaGlcqGGSaalcqWGUbGBcqGGPaqkcqGGSaalaaaaaa@62FF@

D k = ∫ z k z k − 1 f d ( x ) d x + ∫ w k − 1 w k f d ( x ) d x ( k = 2 , ... , n ) . MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaafaqabeqacaaabaGaemiraq0aaSbaaSqaaiabdUgaRbqabaGccqGH9aqpdaWdXaqaaiabdAgaMnaaBaaaleaacqWGKbazaeqaaOGaeiikaGIaemiEaGNaeiykaKIaemizaqMaemiEaGNaey4kaSYaa8qmaeaacqWGMbGzdaWgaaWcbaGaemizaqgabeaakiabcIcaOiabdIha4jabcMcaPiabdsgaKjabdIha4bWcbaGaem4DaC3aaSbaaWqaaiabdUgaRjabgkHiTiabigdaXaqabaaaleaacqWG3bWDdaWgaaadbaGaem4AaSgabeaaa0Gaey4kIipaaSqaaiabdQha6naaBaaameaacqWGRbWAaeqaaaWcbaGaemOEaO3aaSbaaWqaaiabdUgaRjabgkHiTiabigdaXaqabaaaniabgUIiYdaakeaacqGGOaakcqWGRbWAcqGH9aqpcqaIYaGmcqGGSaalcqGGUaGlcqGGUaGlcqGGUaGlcqGGSaalcqWGUbGBcqGGPaqkcqGGUaGlaaaaaa@62EB@

Then the probabilities for tied and concordant pairs become

and the pair consistency probability C can be calculated from equation (1). The combination of cutoff points that maximizes C becomes the solution.

Parametric method for estimating cutoff points and predictive performance

The polychotomization methods proposed in the previous sections have been developed under conditions where the exact distribution of a prognostic or diagnostic factor in a population is known. However, in research practice we work with samples and we need to discuss whether our methods can be applied in situations involving parameter uncertainty. Although some methods were developed for correct estimation of the pair consistency probability C in these situations, including non-parametric ones 222324, none of them addressed the estimation of cutoff points and they can therefore not be applied to our setting.

The challenge we are faced with is that if we repeat the evaluation of the pair consistency probability to find optimal cutoff points, for instance by increasing the possible value of the cutoff point with a certain step, it gives rise to estimation error just like the minimum p-value approach 9101112131415161718 and would mistakenly lead to an optimistic conclusion on the predictive performance of the model in future observations.

It is clear that we need a practical method that does not suffer from this over-estimation bias. In this paper we show that if f_hand f_dcan be transformed to normal distributions, a parametric method provides essentially unbiased estimators of predictive performance and cutoff points.

Our method is based on the following:

a) the assumption that the probability density functions of an independent variable on the healthy and disease groups, f_hand f_d, are both normally distributed or can be transformed to a normal distribution,

b) the estimation of the means and standard deviations of f_hand f_d, m_h, s_h, m_d, and s_dfrom sample data,

c) the localization of the optimal cutoff points based on the estimated distributions f˜h MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGMbGzgaacamaaBaaaleaacqWGObaAaeqaaaaa@2F95@ and f˜d MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGMbGzgaacamaaBaaaleaacqWGKbazaeqaaaaa@2F8D@, and

d) the calculation of the predictive performance based on the estimated cutoff points.

Distributions of the estimators for the cutoff point and the pair consistency probability

If f_hand f_dare both normal and s_h= s_d, then the two curves intersect at x = (m_h+ m_d)/2. The pair consistency probability C takes the maximum value at this point as mentioned earlier. In the case that s_his not equal to s_d, the two curves intersect at the following two points: