The PathBoost algorithm
There are different response types for predictive models built from microarray data, the two most prominent being binary responses, employed, e.g., when classification of tumors is wanted, and time-to-event responses when prediction of survival is wanted. Our proposal for incorporating pathway information is based on likelihood-based boosting 151617. It is therefore suitable for all settings where parameter estimation can be performed by maximization of a likelihood via Newton-Raphson steps. For generalized linear models, the response, which might be continuous, binary or a counting response, is taken to be from an exponential family. Given observations (yi, xi), i = 1,..., n, with response yi and covariate vector xi = (xi1,..., xip)', the structural part of such models is
E(yi|xi) = h(ηi),
where h is a known link function and ηi is the linear predictor
η
i
=
β
i
n
t
e
r
+
x
′
i
β
,
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaeq4TdG2aaSbaaSqaaiabdMgaPbqabaGccqGH9aqpcqaHYoGydaWgaaWcbaGaemyAaKMaemOBa4MaemiDaqNaemyzauMaemOCaihabeaakiabgUcaRiqbdIha4zaafaWaaSbaaSqaaiabdMgaPbqabaGccqaHYoGycqGGSaalaaa@3FA7@
with intercept parameter βinter and parameter vector β = (β1,..., βp)', which are estimated by maximization of the log-likelihood l(β) (see e.g. 18 for more details).
In a time-to-event setting, observations (ti, δi, xi), i = 1,..., n, typically comprise of an observed time ti, a censoring indicator δi, that takes value 1 if the observed time is the time of the event of interest and value 0 if it is the time of censoring, and a covariate vector xi. Due to censoring, direct modeling of ti as a continuous response is problematic. Models for the hazard λ(t|xi), i.e., the instantaneous risk of having an event at time t, given the covariate information, are preferred.
The Cox proportional hazards model has the form
λ(t|xi) = λ0(t) exp(ηi),
where λ0(t) is an unspecified baseline hazards and ηi is a linear predictor of the form
η
i
=
x
′
i
β
,
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaeq4TdG2aaSbaaSqaaiabdMgaPbqabaGccqGH9aqpcuWG4baEgaqbamaaBaaaleaacqWGPbqAaeqaaOGaeqOSdiMaeiilaWcaaa@35FD@
with parameter vector β. Estimation of β is performed by maximizing the partial log-likelihood
l
(
β
)
=
∑
i
=
1
n
δ
i
(
η
i
−
log
(
∑
k
=
1
n
I
(
t
i
≤
t
k
)
exp
(
η
k
)
)
)
,
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaemiBaWMaeiikaGIaeqOSdiMaeiykaKIaeyypa0ZaaabCaeaacqaH0oazdaWgaaWcbaGaemyAaKgabeaakmaabmaabaGaeq4TdG2aaSbaaSqaaiabdMgaPbqabaGccqGHsislcyGGSbaBcqGGVbWBcqGGNbWzdaqadaqaamaaqahabaGaemysaKKaeiikaGIaemiDaq3aaSbaaSqaaiabdMgaPbqabaGccqGHKjYOcqWG0baDdaWgaaWcbaGaem4AaSgabeaakiabcMcaPiGbcwgaLjabcIha4jabcchaWjabcIcaOiabeE7aOnaaBaaaleaacqWGRbWAaeqaaOGaeiykaKcaleaacqWGRbWAcqGH9aqpcqaIXaqmaeaacqWGUbGBa0GaeyyeIuoaaOGaayjkaiaawMcaaaGaayjkaiaawMcaaiabcYcaSaWcbaGaemyAaKMaeyypa0JaeGymaedabaGaemOBa4ganiabggHiLdaaaa@62FC@
where I( ) is an indicator function that takes value 1 if its argument is true and value 0 otherwise, avoiding estimation of the baseline hazard.
Componentwise likelihood-based boosting
The basic idea of boosting is to fit several models to the data in a stepwise manner. In each boosting step, a new model is fitted, which gives larger weight to those observations that were fitted poorly in the previous boosting steps 19. All individual fits are then combined into one overall model. It has been recognized that this procedure is in specific settings equivalent to gradient descent in function space 20, which in turn is equivalent to repeated fitting of residuals for the continuous response case with squared error loss function 14.
In 15, the latter idea is extended to generalized linear models by incorporating the previous boosting steps as an offset into the linear predictor ηi. In 16, a similar approach for boosting estimation of the Cox proportional hazards model is suggested. The basic likelihood-based boosting algorithm is given in the following for both types of models.
Starting with parameter estimate β^0
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGafqOSdiMbaKaadaWgaaWcbaGaeGimaadabeaaaaa@2EA0@ = (0,...,0), in each of k, k = 1,..., M, boosting steps, for each covariate xij, j = 1,..., p, candidate models with linear predictor
η
i
j
,
k
=
η
^
i
,
k
−
1
+
γ
j
,
k
x
i
j
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaeq4TdG2aaSbaaSqaaiabdMgaPjabdQgaQjabcYcaSiabdUgaRbqabaGccqGH9aqpcuaH3oaAgaqcamaaBaaaleaacqWGPbqAcqGGSaalcqWGRbWAcqGHsislcqaIXaqmaeqaaOGaey4kaSIaeq4SdC2aaSbaaSqaaiabdQgaQjabcYcaSiabdUgaRbqabaGccqWG4baEdaWgaaWcbaGaemyAaKMaemOAaOgabeaaaaa@4623@
are fitted by estimating parameters γj, k. The offset η^i,k−1
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGafq4TdGMbaKaadaWgaaWcbaGaemyAaKMaeiilaWIaem4AaSMaeyOeI0IaeGymaedabeaaaaa@3334@ incorporates the information from the previous boosting steps, i.e.,
η
^
i
,
k
−
1
=
x
′
i
β
^
k
−
1
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGafq4TdGMbaKaadaWgaaWcbaGaemyAaKMaeiilaWIaem4AaSMaeyOeI0IaeGymaedabeaakiabg2da9iqbdIha4zaafaWaaSbaaSqaaiabdMgaPbqabaGccuaHYoGygaqcamaaBaaaleaacqWGRbWAcqGHsislcqaIXaqmaeqaaaaa@3CC1@
for the Cox model and
η
^
i
,
k
−
1
=
β
^
i
n
t
e
r
+
x
′
i
β
^
k
−
1
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGafq4TdGMbaKaadaWgaaWcbaGaemyAaKMaeiilaWIaem4AaSMaeyOeI0IaeGymaedabeaakiabg2da9iqbek7aIzaajaWaaSbaaSqaaiabdMgaPjabd6gaUjabdsha0jabdwgaLjabdkhaYbqabaGccqGHRaWkcuWG4baEgaqbamaaBaaaleaacqWGPbqAaeqaaOGafqOSdiMbaKaadaWgaaWcbaGaem4AaSMaeyOeI0IaeGymaedabeaaaaa@467B@
for generalized linear models. The intercept parameter β^inter
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGafqOSdiMbaKaadaWgaaWcbaGaemyAaKMaemOBa4MaemiDaqNaemyzauMaemOCaihabeaaaaa@34A3@ is updated before each boosting step by fitting an intercept-only model.
For estimation of the γj, ks, a penalized log-likelihood criterion
l
p
e
n
(
γ
j
,
k
)
=
l
(
γ
j
,
k
)
+
1
2
λ
j
,
k
γ
j
,
k
2
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaemiBaW2aaSbaaSqaaiabdchaWjabdwgaLjabd6gaUbqabaGccqGGOaakcqaHZoWzdaWgaaWcbaGaemOAaOMaeiilaWIaem4AaSgabeaakiabcMcaPiabg2da9iabdYgaSjabcIcaOiabeo7aNnaaBaaaleaacqWGQbGAcqGGSaalcqWGRbWAaeqaaOGaeiykaKIaey4kaSscfa4aaSaaaeaacqaIXaqmaeaacqaIYaGmaaGccqaH7oaBdaWgaaWcbaGaemOAaOMaeiilaWIaem4AaSgabeaakiabeo7aNnaaDaaaleaacqWGQbGAcqGGSaalcqWGRbWAaeaacqaIYaGmaaaaaa@51EC@
is employed, where λj, k is a penalty parameter that determines the size of the boosting steps. Typically, the same value of λj, k = λ is employed for all covariates and all boosting steps. As the number of boosting steps M, which can, e.g., be determined by cross-validation, is the more important tuning parameter, the penalty parameter λ is chosen only very coarsely, such that the resulting number of boosting steps is not too small (say larger than 50).
Using score function U(γ) = ∂l(γ)/∂γ and information matrix I(γ) = -∂2l(γ)/∂2γ, more specifically the scalar values Uj, k = U(0) and Ij, k = I(0), we employ Newton-Raphson steps, resulting in estimates
γ
^
j
,
k
=
U
j
,
k
I
j
,
k
+
λ
j
,
k
.
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGafq4SdCMbaKaadaWgaaWcbaGaemOAaOMaeiilaWIaem4AaSgabeaakiabg2da9KqbaoaalaaabaGaemyvau1aaSbaaeaacqWGQbGAcqGGSaalcqWGRbWAaeqaaaqaaiabdMeajnaaBaaabaGaemOAaOMaeiilaWIaem4AaSgabeaacqGHRaWkcqaH7oaBdaWgaaqaaiabdQgaQjabcYcaSiabdUgaRbqabaaaaOGaeiOla4caaa@4459@
This is based on only one Newton-Raphson step, as further refinements can potentially be performed in later boosting steps.
The estimate γ^j∗,k
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGafq4SdCMbaKaadaWgaaWcbaGaemOAaO2aaWbaaWqabeaacqGHxiIkaaWccqGGSaalcqWGRbWAaeqaaaaa@327C@ for the covariate with index j* which improves the fit the most (in terms of log-likelihood for generalized linear models or according to the penalized score statistic Uj,k2
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaemyvau1aa0baaSqaaiabdQgaQjabcYcaSiabdUgaRbqaaiabikdaYaaaaaa@31C3@/(Ij, k + λj, k) for the Cox model) is then used to update the elements of the overall parameter vector via
β
^
k
,
j
=
{
β
^
k
−
1
,
j
+
γ
^
j
∗
,
k
for
j
=
j
∗
β
^
k
−
1
,
j
otherwise
.
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGafqOSdiMbaKaadaWgaaWcbaGaem4AaSMaeiilaWIaemOAaOgabeaakiabg2da9maaceaabaqbaeaabiGaaaqaaiqbek7aIzaajaWaaSbaaSqaaiabdUgaRjabgkHiTiabigdaXiabcYcaSiabdQgaQbqabaGccqGHRaWkcuaHZoWzgaqcamaaBaaaleaacqWGQbGAdaahaaadbeqaaiabgEHiQaaaliabcYcaSiabdUgaRbqabaaakeaacqqGMbGzcqqGVbWBcqqGYbGCcqqGGaaicqWGQbGAcqGH9aqpcqWGQbGAdaahaaWcbeqaaiabgEHiQaaaaOqaaiqbek7aIzaajaWaaSbaaSqaaiabdUgaRjabgkHiTiabigdaXiabcYcaSiabdQgaQbqabaaakeaacqqGVbWBcqqG0baDcqqGObaAcqqGLbqzcqqGYbGCcqqG3bWDcqqGPbqAcqqGZbWCcqqGLbqzaaGaeiOla4cacaGL7baaaaa@614F@
This componentwise boosting approach results in sparse fits, i.e., where many elements of the estimated parameter vector are equal to zero.
One of the advantages of likelihood-based boosting is that it is very easy to incorporate mandatory, unpenalized covariates (see 16, for example). This is useful when clinical covariates have to be incorporated in addition to microarray features, in order to compare the resulting model fit to a purely clinical model. The clinical covariates are then added to the linear predictor ηi, and their coefficients are updated in or after every boosting step, but they do not enter into the penalty term.
Incorporating pathway information
The sparseness of the fits, resulting from approaches such as the Lasso or componentwise boosting, is a desirable property in settings with many microarray features, as it potentially results in a short list of genes, that are deemed influential. It can, however, also have a negative effect on interpretability. For example, if the level of activity of (parts of) a specific pathway is related to the response, the microarray features associated with that pathway will be highly correlated and have similar predictive power. However, sparse fitting techniques will probably pick out only one of the features. This makes it difficult to identify the underlying pathway. Also, model fits might be less stable when relying only on one measurement instead of several features.
For discouraging selection of only single microarray features associated with a pathway, we suggest to increase the penalty λj*, l, l > k, used for a specific covariate xij*, after it has been selected in boosting step k. This decreases the size of the boosting steps for this covariate and makes it less likely that this covariate will be selected in future boosting steps. In turn, the penalties for the microarray features that belong to genes that are directly connected in the respective pathway are decreased, making it more likely that they will be selected in future steps.
This approach requires specification of two rules, one for increasing the penalty of a selected covariate and one for decreasing the penalties for connected covariates. In the following, we provide such rules for penalty updates, which, in combination with componentwise likelihood-based boosting, constitute the PathBoost algorithm.
Increasing the penalty for a selected covariate
In order to provide a rule for penalty updates, a common metric for all covariates is needed. Therefore, we quantify the size of the boosting step k, performed for a covariate with index j* that has been selected in this step, by considering the estimate γ^j∗,k
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGafq4SdCMbaKaadaWgaaWcbaGaemOAaO2aaWbaaWqabeaacqGHxiIkaaWccqGGSaalcqWGRbWAaeqaaaaa@327C@ relative to the estimate
γ
^
j
∗
,
k
u
n
p
e
n
=
U
j
∗
,
k
/
I
j
∗
,
k
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGafq4SdCMbaKaadaqhaaWcbaGaemOAaO2aaWbaaWqabeaacqGHxiIkaaWccqGGSaalcqWGRbWAaeaacqWG1bqDcqWGUbGBcqWGWbaCcqWGLbqzcqWGUbGBaaGccqGH9aqpcqWGvbqvdaWgaaWcbaGaemOAaO2aaWbaaWqabeaacqGHxiIkaaWccqGGSaalcqWGRbWAaeqaaOGaei4la8IaemysaK0aaSbaaSqaaiabdQgaQnaaCaaameqabaGaey4fIOcaaSGaeiilaWIaem4AaSgabeaaaaa@47F2@
obtained from unpenalized estimation, i.e., for λj*, k = 0. The step-size factor νj*, k then is given by
ν
j
∗
,
k
=
γ
^
j
∗
,
k
γ
^
j
∗
,
k
u
n
p
e
n
=
I
j
∗
,
k
I
j
∗
,
k
+
λ
j
∗
,
k
.
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaeqyVd42aaSbaaSqaaiabdQgaQnaaCaaameqabaGaey4fIOcaaSGaeiilaWIaem4AaSgabeaakiabg2da9KqbaoaalaaabaGafq4SdCMbaKaadaWgaaqaaiabdQgaQnaaCaaabeqaaiabgEHiQaaacqGGSaalcqWGRbWAaeqaaaqaaiqbeo7aNzaajaWaa0baaeaacqWGQbGAdaahaaqabeaacqGHxiIkaaGaeiilaWIaem4AaSgabaGaemyDauNaemOBa4MaemiCaaNaemyzauMaemOBa4gaaaaakiabg2da9KqbaoaalaaabaGaemysaK0aaSbaaeaacqWGQbGAdaahaaqabeaacqGHxiIkaaGaeiilaWIaem4AaSgabeaaaeaacqWGjbqsdaWgaaqaaiabdQgaQnaaCaaabeqaaiabgEHiQaaacqGGSaalcqWGRbWAaeqaaiabgUcaRiabeU7aSnaaBaaabaGaemOAaO2aaWbaaeqabaGaey4fIOcaaiabcYcaSiabdUgaRbqabaaaaOGaeiOla4caaa@5E4F@
For incorporating pathway information, we suggest to decrease the step-size factor for a selected covariate by a constant step-size modification factor 0 <csmf ≤ 1. So, after the covariate with index j* has been selected in boosting step k, the new step-size factor for further boosting steps l > k becomes
νj*, l = csmf · νj*, k,
implying a penalty increase via
λ
j
∗
,
l
=
(
1
c
s
m
f
−
1
)
I
j
∗
,
l
+
λ
j
∗
,
k
c
s
m
f
.
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaeq4UdW2aaSbaaSqaaiabdQgaQnaaCaaameqabaGaey4fIOcaaSGaeiilaWIaemiBaWgabeaakiabg2da9maabmaajuaGbaWaaSaaaeaacqaIXaqmaeaacqWGJbWydaWgaaqaaiabdohaZjabd2gaTjabdAgaMbqabaaaaiabgkHiTiabigdaXaGccaGLOaGaayzkaaGaemysaK0aaSbaaSqaaiabdQgaQnaaCaaameqabaGaey4fIOcaaSGaeiilaWIaemiBaWgabeaakiabgUcaRKqbaoaalaaabaGaeq4UdW2aaSbaaeaacqWGQbGAdaahaaqabeaacqGHxiIkaaGaeiilaWIaem4AaSgabeaaaeaacqWGJbWydaWgaaqaaiabdohaZjabd2gaTjabdAgaMbqabaaaaOGaeiOla4caaa@530C@
For computational simplicity, we will use a fixed value of Ij*, k+1 instead of the flexible term Ij*, l in this penalty update rule. This means that the new penalty for a covariate can be calculated immediately after it has been selected in a boosting step and that the penalty stays the same until the covariate, or a covariate that is connected to it, is selected again.
Decreasing the penalty for connected covariates
If the penalty for a covariate xij* is increased, and it is then selected again in a later boosting step, the explained variability due to this covariate and the pathways it belongs to will be decreased. To maintain the amount of variability explained by a pathway, the loss in explained variability for covariate xij* is distributed to related covariates, e.g., to covariates that are connected to covariate xij* in the pathway. The amount of potentially lost explained variability, that is to be distributed after a boosting step therefore has to be quantified. A proposal for this is provided in the following.
If k is the first boosting step where covariate xij* is selected, then the unpenalized estimate γ^j∗,kunpen
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGafq4SdCMbaKaadaqhaaWcbaGaemOAaO2aaWbaaWqabeaacqGHxiIkaaWccqGGSaalcqWGRbWAaeaacqWG1bqDcqWGUbGBcqWGWbaCcqWGLbqzcqWGUbGBaaaaaa@3976@ (obtained with λj*, k = 0) will be approximately equal to the (unpenalized) maximum likelihood estimate γ^j∗ml
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGafq4SdCMbaKaadaqhaaWcbaGaemOAaO2aaWbaaWqabeaacqGHxiIkaaaaleaacqWGTbqBcqWGSbaBaaaaaa@3302@ obtained from standard non-boosting estimation. As the relative step size, not realized due to penalized estimation, in boosting step k is given by 1 - νj*, k, for boosting step k + 1, the unpenalized estimate γ^j∗,k+1unpen
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGafq4SdCMbaKaadaqhaaWcbaGaemOAaO2aaWbaaWqabeaacqGHxiIkaaWccqGGSaalcqWGRbWAcqGHRaWkcqaIXaqmaeaacqWG1bqDcqWGUbGBcqWGWbaCcqWGLbqzcqWGUbGBaaaaaa@3B48@ will be approximately equal to
γ
^
j
∗
,
k
+
1
u
n
p
e
n
≈
(
1
−
ν
j
∗
,
k
)
γ
^
j
∗
m
l
.
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGafq4SdCMbaKaadaqhaaWcbaGaemOAaO2aaWbaaWqabeaacqGHxiIkaaWccqGGSaalcqWGRbWAcqGHRaWkcqaIXaqmaeaacqWG1bqDcqWGUbGBcqWGWbaCcqWGLbqzcqWGUbGBaaGccqGHijYUcqGGOaakcqaIXaqmcqGHsislcqaH9oGBdaWgaaWcbaGaemOAaO2aaWbaaWqabeaacqGHxiIkaaWccqGGSaalcqWGRbWAaeqaaOGaeiykaKIafq4SdCMbaKaadaqhaaWcbaGaemOAaO2aaWbaaWqabeaacqGHxiIkaaaaleaacqWGTbqBcqWGSbaBaaGccqGGUaGlaaa@4FAD@
Thus, the penalized estimate γ^j∗,k+1
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGafq4SdCMbaKaadaWgaaWcbaGaemOAaO2aaWbaaWqabeaacqGHxiIkaaWccqGGSaalcqWGRbWAcqGHRaWkcqaIXaqmaeqaaaaa@344E@ will be
γ
^
j
∗
,
k
+
1
=
ν
j
∗
,
k
+
1
⋅
γ
^
j
∗
,
k
+
1
u
n
p
e
n
≈
ν
j
∗
,
k
+
1
(
1
−
ν
j
∗
,
k
)
γ
^
j
∗
m
l
.
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGafq4SdCMbaKaadaWgaaWcbaGaemOAaO2aaWbaaWqabeaacqGHxiIkaaWccqGGSaalcqWGRbWAcqGHRaWkcqaIXaqmaeqaaOGaeyypa0JaeqyVd42aaSbaaSqaaiabdQgaQnaaCaaameqabaGaey4fIOcaaSGaeiilaWIaem4AaSMaey4kaSIaeGymaedabeaakiabgwSixlqbeo7aNzaajaWaa0baaSqaaiabdQgaQnaaCaaameqabaGaey4fIOcaaSGaeiilaWIaem4AaSMaey4kaSIaeGymaedabaGaemyDauNaemOBa4MaemiCaaNaemyzauMaemOBa4gaaOGaeyisISRaeqyVd42aaSbaaSqaaiabdQgaQnaaCaaameqabaGaey4fIOcaaSGaeiilaWIaem4AaSMaey4kaSIaeGymaedabeaakiabcIcaOiabigdaXiabgkHiTiabe27aUnaaBaaaleaacqWGQbGAdaahaaadbeqaaiabgEHiQaaaliabcYcaSiabdUgaRbqabaGccqGGPaqkcuaHZoWzgaqcamaaDaaaleaacqWGQbGAdaahaaadbeqaaiabgEHiQaaaaSqaaiabd2gaTjabdYgaSbaakiabc6caUaaa@6C88@
The approximate fraction πj,(m) of the maximum likelihood estimate that has been realized for covariate xij in the mth boosting step, where this covariate has been selected, then is
π
j
,
(
1
)
=
ν
j
,
(
1
)
π
j
,
(
2
)
=
ν
j
,
(
1
)
+
(
1
−
ν
j
,
(
1
)
)
ν
j
,
(
2
)
⋯
π
j
,
(
m
)
=
π
j
,
(
m
−
1
)
+
(
1
−
π
j
,
(
m
−
1
)
)
ν
j
,
(
m
)
.
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaqbaeaabqWaaaaabaGaeqiWda3aaSbaaSqaaiabdQgaQjabcYcaSiabcIcaOiabigdaXiabcMcaPaqabaaakeaacqGH9aqpaeaacqaH9oGBdaWgaaWcbaGaemOAaOMaeiilaWIaeiikaGIaeGymaeJaeiykaKcabeaaaOqaaiabec8aWnaaBaaaleaacqWGQbGAcqGGSaalcqGGOaakcqaIYaGmcqGGPaqkaeqaaaGcbaGaeyypa0dabaGaeqyVd42aaSbaaSqaaiabdQgaQjabcYcaSiabcIcaOiabigdaXiabcMcaPaqabaGccqGHRaWkcqGGOaakcqaIXaqmcqGHsislcqaH9oGBdaWgaaWcbaGaemOAaOMaeiilaWIaeiikaGIaeGymaeJaeiykaKcabeaakiabcMcaPiabe27aUnaaBaaaleaacqWGQbGAcqGGSaalcqGGOaakcqaIYaGmcqGGPaqkaeqaaaGcbaaabaGaeS47IWeabaaabaGaeqiWda3aaSbaaSqaaiabdQgaQjabcYcaSiabcIcaOiabd2gaTjabcMcaPaqabaaakeaacqGH9aqpaeaacqaHapaCdaWgaaWcbaGaemOAaOMaeiilaWIaeiikaGIaemyBa0MaeyOeI0IaeGymaeJaeiykaKcabeaakiabgUcaRiabcIcaOiabigdaXiabgkHiTiabec8aWnaaBaaaleaacqWGQbGAcqGGSaalcqGGOaakcqWGTbqBcqGHsislcqaIXaqmcqGGPaqkaeqaaOGaeiykaKIaeqyVd42aaSbaaSqaaiabdQgaQjabcYcaSiabcIcaOiabd2gaTjabcMcaPaqabaGccqGGUaGlaaaaaa@84A8@
Let now j1 be the index of a covariate that has been selected in boosting step k and j2 be the index of the covariate to which a potential loss in explained variability is to be transferred. There is a potential loss that is incurred for xij1
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaemiEaG3aaSbaaSqaaiabdMgaPjabdQgaQnaaBaaameaacqaIXaqmaeqaaaWcbeaaaaa@315A@ in a future boosting step l by employing a penalty that is updated via (1), with corresponding step-size factor νj1,l
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaeqyVd42aaSbaaSqaaiabdQgaQnaaBaaameaacqaIXaqmaeqaaSGaeiilaWIaemiBaWgabeaaaaa@327F@, instead of not modifying the penalty, i.e., keeping the step-size factor νj1,k
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaeqyVd42aaSbaaSqaaiabdQgaQnaaBaaameaacqaIXaqmaeqaaSGaeiilaWIaem4AaSgabeaaaaa@327D@. In terms of the fraction of the maximum likelihood estimate this loss is given by
(
1
−
π
j
1
,
k
)
(
ν
j
1
,
k
−
ν
j
1
,
l
)
.
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaeiikaGIaeGymaeJaeyOeI0IaeqiWda3aaSbaaSqaaiabdQgaQnaaBaaameaacqaIXaqmaeqaaSGaeiilaWIaem4AaSgabeaakiabcMcaPiabcIcaOiabe27aUnaaBaaaleaacqWGQbGAdaWgaaadbaGaeGymaedabeaaliabcYcaSiabdUgaRbqabaGccqGHsislcqaH9oGBdaWgaaWcbaGaemOAaO2aaSbaaWqaaiabigdaXaqabaWccqGGSaalcqWGSbaBaeqaaOGaeiykaKIaeiOla4caaa@4752@
The aim is now to choose the penalty λj2,l
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaeq4UdW2aaSbaaSqaaiabdQgaQnaaBaaameaacqaIYaGmaeqaaSGaeiilaWIaemiBaWgabeaaaaa@327D@, or correspondingly the step-size factor νj2,l
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaeqyVd42aaSbaaSqaaiabdQgaQnaaBaaameaacqaIYaGmaeqaaSGaeiilaWIaemiBaWgabeaaaaa@3281@, for the covariate with index j2 for a future boosting step l (compared to step k), such that the loss for covariate xij1
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaemiEaG3aaSbaaSqaaiabdMgaPjabdQgaQnaaBaaameaacqaIXaqmaeqaaaWcbeaaaaa@315A@ is compensated by covariate xij2
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaemiEaG3aaSbaaSqaaiabdMgaPjabdQgaQnaaBaaameaacqaIYaGmaeqaaaWcbeaaaaa@315C@. Equating
(
1
−
π
j
2
,
k
)
ν
j
2
,
l
−
(
1
−
π
j
2
,
k
)
ν
j
2
,
k
=
(
1
−
π
j
1
,
k
)
(
ν
j
1
,
k
−
ν
j
1
,
l
)
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaeiikaGIaeGymaeJaeyOeI0IaeqiWda3aaSbaaSqaaiabdQgaQnaaBaaameaacqaIYaGmaeqaaSGaeiilaWIaem4AaSgabeaakiabcMcaPiabe27aUnaaBaaaleaacqWGQbGAdaWgaaadbaGaeGOmaidabeaaliabcYcaSiabdYgaSbqabaGccqGHsislcqGGOaakcqaIXaqmcqGHsislcqaHapaCdaWgaaWcbaGaemOAaO2aaSbaaWqaaiabikdaYaqabaWccqGGSaalcqWGRbWAaeqaaOGaeiykaKIaeqyVd42aaSbaaSqaaiabdQgaQnaaBaaameaacqaIYaGmaeqaaSGaeiilaWIaem4AaSgabeaakiabg2da9iabcIcaOiabigdaXiabgkHiTiabec8aWnaaBaaaleaacqWGQbGAdaWgaaadbaGaeGymaedabeaaliabcYcaSiabdUgaRbqabaGccqGGPaqkcqGGOaakcqaH9oGBdaWgaaWcbaGaemOAaO2aaSbaaWqaaiabigdaXaqabaWccqGGSaalcqWGRbWAaeqaaOGaeyOeI0IaeqyVd42aaSbaaSqaaiabdQgaQnaaBaaameaacqaIXaqmaeqaaSGaeiilaWIaemiBaWgabeaakiabcMcaPaaa@6A5B@
results in an update for the step-size factor
ν
j
2
,
l
=
ν
j
2
,
k
+
1
−
π
j
1
,
k
1
−
π
j
2
,
k
(
ν
j
1
,
k
−
ν
j
1
,
l
)
.
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaeqyVd42aaSbaaSqaaiabdQgaQnaaBaaameaacqaIYaGmaeqaaSGaeiilaWIaemiBaWgabeaakiabg2da9iabe27aUnaaBaaaleaacqWGQbGAdaWgaaadbaGaeGOmaidabeaaliabcYcaSiabdUgaRbqabaGccqGHRaWkjuaGdaWcaaqaaiabigdaXiabgkHiTiabec8aWnaaBaaabaGaemOAaO2aaSbaaeaacqaIXaqmaeqaaiabcYcaSiabdUgaRbqabaaabaGaeGymaeJaeyOeI0IaeqiWda3aaSbaaeaacqWGQbGAdaWgaaqaaiabikdaYaqabaGaeiilaWIaem4AaSgabeaaaaGccqGGOaakcqaH9oGBdaWgaaWcbaGaemOAaO2aaSbaaWqaaiabigdaXaqabaWccqGGSaalcqWGRbWAaeqaaOGaeyOeI0IaeqyVd42aaSbaaSqaaiabdQgaQnaaBaaameaacqaIXaqmaeqaaSGaeiilaWIaemiBaWgabeaakiabcMcaPiabc6caUaaa@5DD8@
This implies a decrease of the penalty parameter λj2,k
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaeq4UdW2aaSbaaSqaaiabdQgaQnaaBaaameaacqaIYaGmaeqaaSGaeiilaWIaem4AaSgabeaaaaa@327B@ via
λ
j
2
,
l
=
(
1
−
π
j
2
,
k
)
I
j
2
,
l
(
1
−
c
s
m
f
)
(
1
−
π
j
1
,
k
)
I
j
1
,
l
I
j
1
,
l
+
λ
j
1
,
k
+
(
1
−
π
j
2
,
k
)
I
j
2
,
l
I
j
2
,
l
+
λ
j
2
,
k
−
I
j
2
,
l
.
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaeq4UdW2aaSbaaSqaaiabdQgaQnaaBaaameaacqaIYaGmaeqaaSGaeiilaWIaemiBaWgabeaakiabg2da9KqbaoaalaaabaGaeiikaGIaeGymaeJaeyOeI0IaeqiWda3aaSbaaeaacqWGQbGAdaWgaaqaaiabikdaYaqabaGaeiilaWIaem4AaSgabeaacqGGPaqkcqWGjbqsdaWgaaqaaiabdQgaQnaaBaaabaGaeGOmaidabeaacqGGSaalcqWGSbaBaeqaaaqaaiabcIcaOiabigdaXiabgkHiTiabdogaJnaaBaaabaGaem4CamNaemyBa0MaemOzaygabeaacqGGPaqkdaWcaaqaaiabcIcaOiabigdaXiabgkHiTiabec8aWnaaBaaabaGaemOAaO2aaSbaaeaacqaIXaqmaeqaaiabcYcaSiabdUgaRbqabaGaeiykaKIaemysaK0aaSbaaeaacqWGQbGAdaWgaaqaaiabigdaXaqabaGaeiilaWIaemiBaWgabeaaaeaacqWGjbqsdaWgaaqaaiabdQgaQnaaBaaabaGaeGymaedabeaacqGGSaalcqWGSbaBaeqaaiabgUcaRiabeU7aSnaaBaaabaGaemOAaO2aaSbaaeaacqaIXaqmaeqaaiabcYcaSiabdUgaRbqabaaaaiabgUcaRmaalaaabaGaeiikaGIaeGymaeJaeyOeI0IaeqiWda3aaSbaaeaacqWGQbGAdaWgaaqaaiabikdaYaqabaGaeiilaWIaem4AaSgabeaacqGGPaqkcqWGjbqsdaWgaaqaaiabdQgaQnaaBaaabaGaeGOmaidabeaacqGGSaalcqWGSbaBaeqaaaqaaiabdMeajnaaBaaabaGaemOAaO2aaSbaaeaacqaIYaGmaeqaaiabcYcaSiabdYgaSbqabaGaey4kaSIaeq4UdW2aaSbaaeaacqWGQbGAdaWgaaqaaiabikdaYaqabaGaeiilaWIaem4AaSgabeaaaaaaaOGaeyOeI0IaemysaK0aaSbaaSqaaiabdQgaQnaaBaaameaacqaIYaGmaeqaaSGaeiilaWIaemiBaWgabeaakiabc6caUaaa@914E@
Again, for computational simplicity, we use a fixed value of Ij1,k+1
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaemysaK0aaSbaaSqaaiabdQgaQnaaBaaameaacqaIXaqmaeqaaSGaeiilaWIaem4AaSMaey4kaSIaeGymaedabeaaaaa@33B2@ instead of Ij1,l
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaemysaK0aaSbaaSqaaiabdQgaQnaaBaaameaacqaIXaqmaeqaaSGaeiilaWIaemiBaWgabeaaaaa@31E2@, and a value of Ij2,k+1
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaemysaK0aaSbaaSqaaiabdQgaQnaaBaaameaacqaIYaGmaeqaaSGaeiilaWIaem4AaSMaey4kaSIaeGymaedabeaaaaa@33B4@ instead of Ij2,l
MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaemysaK0aaSbaaSqaaiabdQgaQnaaBaaameaacqaIYaGmaeqaaSGaeiilaWIaemiBaWgabeaaaaa@31E4@ in this update rule. Therefore, the new penalties of connected covariates can be calculated immediately after the boosting step, avoiding recalculation after every boosting step and storage of results from past boosting steps.
As an increase of the penalty via (1) would leave the potential loss in explained variability undistributed for a covariate without connections, the penalty update is only performed for covariates that correspond to genes that have a connection to another gene, with corresponding covariate, in a pathway. For connected genes, however, the question remains whether the total amount should be transferred to every connected covariate or whether the right-hand side of (2) should be divided by the number of connections. As componentwise boosting results in very sparse fits, it can be expected that only few connected covariates will be selected in the remaining boosting steps. It therefore seems to be reasonable to assign the amount to each connected covariate.
While a measure of uncertainty is not available for connections in a pathway in the KEGG pathway database, it might be available from other sources. Such information could easily be incorporated into the PathBoost algorithm by multiplying the right-hand side of (2) by the measure of uncertainty (given that the latter has values between 0 an 1). Also, information on the direction of relations could be incorporated by propagating changes of a penalty only into one direction.
Choice of tuning parameters
The proposed PathBoost algorithm has three flexible parameters: an initial penalty λj,1 = λ, j = 1,..., p, common to all covariates, the number of boosting steps M, and the step-size modification factor csmf. The initial penalty parameter is of minor importance and can be chosen very coarsely. A value that roughly corresponds to initial step-size factors of about 0.01 works very well in our experience. For determining the step-size modification factor csmf, a coarse line search is performed. For each value of csmf, the optimal number of boosting steps is determined by 10-fold cross-validation. Then the value of csmf, which results in the overall maximum of cross-validated (partial) log-likelihood, is chosen.
Simulation study
To evaluate the performance of the PathBoost approach, we perform a small simulation study that is identical, in terms of design, to the study employed in 9. Models for a continuous response are built from p = 2200 covariates. Of these, 200 take the role of transcription factors. The remaining 2000 covariates comprise of blocks of 10 covariates, where the covariates in each block are correlated with one specific transcription factor. The connection information, required for the approach given in 9 and for the PathBoost approach, is chosen such that there is a bidirectional connection between each transcription factor and each of the 10 covariates associated with it.
The true parameter vector in the generating linear model is chosen such that only four transcription factors (and the corresponding blocks of correlated covariates) have an effect on the response. There are six types of generating models with varying size and type of effect.
In Model 1, the true parameters of the covariates that are related to a transcription factor have the same sign as the parameter of the transcription factor itself. This is expected to be favorable for the approach given in 9, as the penalty term employed there penalizes the squared (standardized) differences of parameters. However, for true parameters with opposite sign, this difference will be large, making it rather unlikely that the true values are recovered. Model 2 features such a setting, where in each block of 10 informative covariates, the parameters of three covariates have a sign opposite to that of the associated transcription factor. In 9 it was found that this considerably affected the performance of the approach with an explicit penalty structure. In contrast, we do not expect a performance degradation for the PathBoost approach as it does not rely on differences of parameters.
Model 3 is similar to Model 1, and Model 4 is similar to Model 2, the only difference being a smaller effect of the covariates. Extending the design given in 9, we added two further settings, Model 5 and Model 6, which are based on Model 2 and Model 4 respectively. In these settings, only the first and the third block of informative covariates contain effects with opposite sign. Therefore, only six of a total of 40 informative connected covariates have an effect with a sign opposite to the associated transcription factor.
As a minimal performance reference, an intercept-only model, i.e., a model that does not use any covariate information, is fitted. A more specific performance reference for the PathBoost approach is provided by componentwise likelihood-based boosting without pathway information 15. The main tuning parameter there is the number of boosting steps, which is determined by 10-fold cross-validation. As already suggested, the additional parameter csmf for the PathBoost approach is determined by a coarse line search.
As a performance reference for the approach given in 9, models are fitted by the Lasso 10, which also penalizes the absolute values of the parameters, but does not incorporate pathway information. For both approaches, fitting is performed by the least angle regression technique 21, which allows for fast computation of solutions for a large range of values for the penalty parameter that governs the absolute value term in the penalty. For the Lasso, only the latter has to be chosen, which is done by 10-fold cross-validation. For the approach given in 9, a second penalty parameter is required, which, similar to the PathBoost approach, is determined by a coarse line search.
All approaches are fitted to training sets of size n = 100, and prediction performance is evaluated on a test set of the same size. This is repeated 50 times. Table 1 shows the corresponding mean values and standard errors of the predictive mean squared error.
<p>Table 1</p>
Results of the simulation study.
Model
intercept
Lasso
Li&Li
lik.boost
PathBoost
1
762.5 (14.4)
83.6 (2.6)
42.5 (1.1)
83.4 (2.4)
61.0 (1.7)
2
305.8 (5.1)
91.0 (2.7)
80.8 (1.9)
89.7 (2.7)
64.8 (1.8)
3
215.6 (4.1)
32.6 (0.9)
24.9 (0.8)
32.1 (0.9)
26.5 (0.7)
4
131.1 (2.4)
32.6 (0.9)
29.9 (0.7)
32.5 (0.9)
26.9 (0.7)
5
525.7 (9.9)
87.9 (2.6)
61.6 (1.5)
85.6 (2.2)
62.2 (1.6)
6
171.6 (3.3)
32.9 (0.9)
27.6 (0.7)
32.2 (0.9)
26.9 (0.8)
Predictive mean squared error, mean and standard errors (in parentheses), for an intercept-only model, the Lasso, the pathway-based procedure proposed in 9 (Li&Li), componentwise likelihood-based boosting (lik.boost), and boosting with pathway information (PathBoost) for six types of generating models.
The predictive mean squared error for all approaches is far below that of the intercept-only model, indicating that the prediction problems are very simple. As would be expected, the performance of the Lasso and componentwise boosting is very similar. So, there is no disadvantage of choosing one of the two as a basis for an approach that incorporates pathway information.
The approach given in 9 outperforms the Lasso in all six settings. However, the performance difference is greatly diminished with Models 2 and 4, where several of the parameters of connected covariates have opposite sign. This highlights the difficulties potentially arising from an explicitly specified penalty structure. In contrast, the PathBoost approach is seen to result in a consistent improvement over boosting without pathway information in all settings. As would be expected from the design of the algorithm, the sign of the true parameters does not matter.
Comparing the PathBoost approach to that given in 9, the latter shows better prediction performance for Models 1 and 2, i.e., where its penalty structure matches the sign of the true parameters. However, for Models 3 and 4, where the sign of parameters of connected covariates may be different, the approach given in 9 performs worse. The performance of the two approaches is similar for Models 5 and 6, implying that already a small mismatch in sign information can nullify potential performance advantages gained by explicitly specifying the penalty structure in the approach given in 9.
Application examples
In the following, we investigate the properties of the PathBoost approach in two application examples with microarray survival data, where a Cox proportional hazards model is fitted. When applying a technique for fitting predictive models that incorporates pathway information in a real application setting, there are two objectives. The first is to get better interpretability of the model fit, but the interpretation of a fit will only be credible if the second objective, that of improved prediction performance, is met. For adequately evaluating a potential gain in prediction performance from incorporating pathway information in a time-to-event setting, we employ bootstrap .632+ prediction error curve estimates 222324.
Pathway information is extracted from the KEGG pathway database 1. Similar to 9, we restrict analyses to regulatory pathways, but also include cancer pathways. As a restriction to gene-gene relations would have resulted in a very small number of connections, any genes that are linked by some kind of KEGG relation are considered to be connected.
While the glioblastoma data analyzed in 9 has a time-to-event response, closer inspection showed that the genes which have predictive power are not represented in KEGG pathways. Therefore, an approach focussed on the latter cannot improve over a null model that does not use any microarray information 25. We investigate two other data sets, one from patients with large B-cell lymphoma 26 and a second from patients with ovarian cancer 8.
Diffuse large B-cell lymphoma
The data from patients with diffuse large B-cell lymphoma (DLBCL) has already been used for illustrating prediction error curve techniques 23 and the likelihood-based boosting technique for the Cox proportional hazards model 16, on which the PathBoost approach is based. Details of preprocessing are described there. There are n = 240 observation with p = 7399 microarray features. Only 1281 of the latter could be related to KEGG pathways, based on the information available. To avoid restriction to a (relatively) small number of microarray features and to maintain comparability to previous analyses, also the features not represented in KEGG pathways are considered.
A coarse line search, in combination with 10-fold cross validation, results in selection of a step-size modification factor of csmf = 0.9, which indicates that there might be some predictive pathway information in the data. Use of this factor results in 47 non-zero coefficients. In comparison, application of boosting without pathway information results in only 27 non-zero coefficients. There is an overlap of 20 non-zero coefficients, indicating that seven microarray features are no longer deemed important when pathway information is included, with 27 new features being added to the model.
For checking whether the 27 added features just contain information similar to the seven features not found in the PathBoost fit, we applied componentwise boosting to a data set where the seven features were removed. If the 27 seven features would be a substitute for the seven removed features, some of the former should now be included. However, the resulting model has 20 non-zero coefficients, which all belong to the same covariates as the overlapping coefficients above, i.e., none of the 27 microarray features, identified by PathBoost, are included in the model. The prediction performance decreases (not shown), indicating that the seven microarray features contain information which is useful in combination with componentwise boosting. However, as PathBoost does not utilize these seven microarray features and nevertheless performs better, this underlines that PathBoost results in structurally different model fits.
While in the model, fitted by boosting without pathway information, only two connected microarray features receive non-zero coefficient estimates, PathBoost results in 12 connected microarray features that receive non-zero estimates. This indicates that the fit from the latter algorithm reflects pathway knowledge. The coefficients of connected microarray features have different sign in several instances. As such a constellation did not influence the performance of PathBoost in the simulation study, an impact is also not expected in this application example.
The change in structure of the fitted models is also seen from the coefficient paths, i.e., the parameter estimates plotted against the boosting steps. Figure 1 shows the coefficient paths for boosting without pathway information (left panel) and PathBoost (right panel). While they are rather similar, there are some features with strong effect that appear only in the PathBoost fit (e.g., UNIQIDs 29911 and 27573). As the PathBoost algorithm increases the penalty for a covariate after it has been selected, it could be expected that the estimates are somewhat shrunken compared to the CoxBoost fit. This is seen, e.g., for the microarray features with UNIQIDs 32238 and 32679, which are no longer selected by PathBoost after a certain boosting step, as the penalty for them has become too large. This is different from approaches that use an explicit shrinkage term in the penalized (partial) log-likelihood criterion, as there it would be expected that the whole path is shrunken.
<p>Figure 1</p>
Coefficient paths for the DLBCL data
Coefficient paths for the DLBCL data. Coefficient paths for boosting without pathway information (left panel) and PathBoost (right panel), applied to DLBCL data. The models selected by 10-fold cross validation are indicated by vertical lines. Microarray features common to both models are indicated by solid curves, the others by dotted curves.
![]()
While use of pathway information is seen to have influenced the model fit, interpretation of the latter can only be assumed to be more valid, compared to the fit obtained without pathway information, if prediction performance is also improved. The thick curves in Figure 2 indicate .632+ prediction error curve estimates (based on 100 bootstrap samples of size 0.632n, drawn without replacement). The Kaplan-Meier benchmark (grey curve) that does not use any covariate information is given as a reference. All procedures are seen to improve over the Kaplan-Meier benchmark, where PathBoost (solid curve) seems to have a slight advantage over boosting without pathway information (dashed curve). While the difference is not very large, it nevertheless improves the credibility of the PathBoost fit.
<p>Figure 2</p>
Prediction error curves for the DLBCL data
Prediction error curves for the DLBCL data. Bootstrap .632+ prediction error curve estimates for boosting without pathway information (dashed curves) and PathBoost (solid curves), applied to DLBCL data, without (thick curves) and with clinical covariates (thin curves). The Kaplan-Meier benchmark (grey curve) and a purely clinical model (dotted curve) are given as a reference.
![]()
For 222 patients, a clinical predictor, the International Prognostic Index (IPI), is available. As it is typically of interest how much microarray information can improve over purely clinical models, we include the clinical covariate as a mandatory, unpenalized covariate, as described in 16. The corresponding prediction error curve estimates are indicated by thin curves in Figure 2. The prediction performance of a purely clinical model is indicated by the dotted curve. It is seen that the combined models can improve over the purely clinical model. However, PathBoost (solid curve) can no longer improve over boosting without pathway information (dashed curve). The lack of additional value of pathway information in this setting is also reflected by the step-size modification factor, chosen by a line search, which is csmf = 1. Therefore it seems that, in the present example, pathway information is most useful in describing phenomena that are already reflected by the clinical covariate.
Ovarian cancer
The second data set, to be used for illustration of the PathBoost approach, is from patients with ovarian cancer. The original analysis of this data 8 already showed that there is a connection between pathway activity and survival, where pathway signatures were derived from prior experiments. In contrast, we will investigate whether pathway knowledge derived from the KEGG database can also add to prediction of survival.
For the 133 patients, where time-to-event information is available, we performed preprocessing of the microarray data, using the RMA approach 27, resulting in 21801 microarray features. We restrict analysis to those 4868 features that are related to any of the human KEGG pathways.
The connections between genes, just as for the DLBCL data, are extracted from the regulatory KEGG pathways, including the cancer pathways. The step-size modification factor, selected by a line search in combination with 10-fold cross-validation, then is csmf = 1, i.e., pathway information would not be expected to be useful for prediction of survival. However, when only the connections from the cancer pathways are considered, the resulting factor is csmf = 0.9. This indicates that targeted pathway information might be useful, while use of too many pathways is detrimental to prediction performance. Figure 3 shows bootstrap .632+ prediction error curve estimates for boosting without pathway information (thick dashed curve) and for PathBoost approach (thick solid curve), when considering only the cancer pathways. We also investigate models that incorporate the clinical covariate "tumor stage" as a mandatory unpenalized covariate (thin curves). All models perform considerably better than the Kaplan-Meier benchmark. Just as for the DLBCL data, there is an advantage of PathBoost over boosting without pathway information, albeit a smaller one, indicating usefulness of pathway information for prediction. In contrast to the DLBCL example, PathBoost also performs better when the clinical covariate is included. This indicates that the pathways provide information in addition to the clinical covariate.
<p>Figure 3</p>
Prediction error curves for the ovarian cancer data
Prediction error curves for the ovarian cancer data. Bootstrap .632+ prediction error curve estimates for boosting without pathway information (dashed curves) and PathBoost (solid curves), applied to ovarian cancer data, without (thick curves) and with clinical covariates (thin curves). The Kaplan-Meier benchmark (grey curve) is given as a reference.
![]()