The symbols and notations used in this paper are defined in Table 1 (in the order of appearance).

We consider the problem of minimizing a quadratic cost of a real vector in function of and a real positive semi-definite (PSD) matrix Q, given by

where denotes a convex set. Also, PSD implies that . We will show that many machine learning problems can be cast in form (1) with additional constraints on . In particular, if we restrict , the problem in (1) becomes a Rayleigh quotient and leads to the eigenvalue problem. Now we consider a convex parametric linear combination of a set of p PSD matrices Q _j , given by:

To bound the coefficients θ _j , we restrict that, for example, ||θ _j ||₁ = 1, and (1) can be equivalently rewritten as a min-max problem, given by

To solve (3), we denote , the min-max problem can be formulated in a form of quadraticly constrained linear program (QCLP), given by

The optimal solution in (3) is obtained from the dual variable corresponding to the quadratic constraints in (4). The optimal t* is equivalent to the Chebyshev or L _∞-norm of the vector of quadratic terms, given by:

The L _∞-norm is the upper bound w.r.t. the constraint because

Apparently, suppose the optimal is given, optimizing the L _∞-norm in (5) will pick the single term with the maximal value, and the optimal solution of the coefficients is more likely to be sparse. An alternative solution to (3) is to introduce a different constraint on the coefficients, for example, ||θ _j ||₂ = 1. We thus propose a new extension of the problem in (1), given by

This new extension is analogously solved as a QCLP problem with modified constraints, given by

where . The proof that (8) is the solution of (7) is given in the following theorem.

Theorem 0.1 The QCLP problem in (8) equivalently solves the problem in (7).

Proof Given two vectors {x ₁, ..., x _p }, {y ₁, ..., y _p }, x _j , y _j ∈ ℝ, j = 1, ..., p, the Cauchy-Schwarz inequality states that

with as equivalent form:

Let us denote x _j = θ _j and , (10) becomes

Since ||θ _j ||₂ = 1, (11) is equivalent to

Therefore, given , the additive term is bounded by the L ₂-norm || ||₂.

Moreover, it is easy to prove that when , the parametric combination reaches the upperbound and the equality holds. Optimizing this L ₂-norm results in a non-sparse solution in θ _j . In order to distinguish this from the solution obtained by (3) and (4), we denote it as the L ₂-norm approach. It can also easily be seen (not shown here) that the L ₁-norm approach is simply averaging the quadratic terms with uniform coefficients.

The L ₂-norm bound is also generalizable to any positive real number n ≥ 1, defined as L _n -norm MKL. Recently, the similar topic is also investigated by 9 and a solution is proposed to solve the primal MKL problem. In this paper, we will show that our primal-dual interpretation of MKL is also extendable to the n-norm. Let us assume that is regularized by the L _m -norm as || || _m = 1, then the L _m -norm extension of equation (7) is given by

In the following theorem, we prove that (13) can be equivalently solved as a QCLP problem, given by

where and the constraint is in L _n -norm, moreover, . The problem in (14) is convex and can be solved by cvx toolbox 10 11 .

Theorem 0.2 If the coefficient vector is regularized by a L _m -norm in (13), the problem can be solved as a convex programming problem in (14) with L _n -norm constraint. Moreover, .

Proof We generalize the Cauchy-Schwarz inequality to Hölder's inequality. Let m, n > 1 be two numbers that satisfy . Then

Let us denote x _j = θ _j and , (2) becomes

Since || || _m = 1, therefore the term can be omitted in the equation, so (3) is equivalent to

Due to the condition that , so , we prove that with the L _m -norm constraint posed on , the additive multiple kernel term is bounded by the L _n -norm of the vector . Moreover, we have .

In this section, we have explained the L _∞, L ₁, L ₂, and L _n -norm approaches to extend the basic problem in (1) to multiple matrices Q _j . These approaches differed mainly on the constraints applied on the coefficients. To clarify the difference of notations used in this paper with the common interpretations of L ₁ and L ₂ regularization on , we illustrate the mapping of our L _∞, L ₁, L ₂, and L _n notations between the common interpretations of coefficient regularization. As shown in Table 2, the notations used in this paper are interpreted in the dual space and are equivalent to regularization of kernel coefficients in the primal space. The advantage of dual space interpretation is that we can easily extend the analogue solution to various machine learning algorithms. To keep the discussion concise, we will from now on mainly focus on comparing the L _∞, L ₁ and L ₂ in the dual problems and present the solutions in the dual space.

Next, we will investigate several concrete kernel fusion algorithms and will propose the corresponding L ₂ solutions.

The primal problem of one class SVM (1-SVM) is defined by Tax and Duin 12 and Schölkopf et al. 13 as

where is the norm vector of the separating hyperplane, are the training samples, ν is the regularization constant penalizing outliers in the training samples, ϕ(·) denotes the feature map, ρ is a bias term, ξ _i are slack variables, and N is the number of training samples. Taking the conditions for optimality from the Lagrangian, one obtains the dual problem, given by:

where α _i are dual variables, K represents the kernel matrix obtained by the inner product between any pair of samples specified by a kernel function . To incorporate multiple kernels in (19), De Bie et al. proposed a solution 7 with the dual problem formulated as

where p is the number of data sources and K _j is the j-th kernel matrix. The formulation exactly corresponds to the L _∞ solution of the problem defined in the previous section (the PSD constraint is implied in the kernel matrix) with additional constraints imposed on . The optimal coefficients θ _j are used to combine multiple kernels as

and the ranking function is given by

where Ω _N is the combined kernel of training data , i = 1, ..., N, is the test data point to be ranked, is the kernel function applied on test data and training data, is the dual variable solved as (20). De Bie et al. applied the method in the application of disease gene prioritization, where multiple genomic data sources are combined to rank a large set of test genes using the 1-SVM model trained from a small set of training genes which are known to be relevant for certain diseases. The L _∞ formulation in their approach yields a sparse solution when integrating genomic data sources (see Figure 2 of 7 ). To avoid this disadvantage, they proposed a regularization method by restricting the minimal boundary on the kernel coefficients, notated as θ _min , to ensure the minimal contribution of each genomic data source to be θ _min /p. According to their experiments, the regularized solution performed best, being significantly better than the sparse integration and the average combination of kernels.

Instead of setting the ad hoc parameter θ _min , one can also straightforwardly propose an L ₂-norm approach to solve the identical problem, given by

where . The problem above is a QCLP problem and can be solved by conic optimization solvers such as Sedumi 14 . In (23), the first constraint represents a Lorentz cone and the second constraint corresponds to p number of rotated Lorentz cones (R cones). The optimal kernel coefficients θ _j correspond to the dual variables of the R cones with ||θ||₂ = 1. In this L ₂-norm approach, the integrated kernel Ω is combined by different and the same scoring function as in (22) is applied on the different solutions of and Ω.

The notion of MKL is originally proposed in a binary SVM classification, where the primal objective is given by

where are data samples, ϕ(·) is the feature map, y _i are class labels, C > 0 is a positive regularization parameter, ξ _i are slack variables, is the norm vector of the separating hyperplane, and b is the bias. This problem is convex and can be solved as a dual problem, given by

where are the dual variables, Y = diag(y ₁, ..., y _N ), K is the kernel matrix, and C is the upperbound of the box constraint on the dual variables. To incorporate multiple kernels in (25), Lanckriet et al. 6 4 and Bach et al. 5 proposed a multiple kernel learning (MKL) problem as follows:

where p is the number of kernels. (26) optimizes the L _∞-norm of the set of kernel quadratic terms. Based on the previous discussions, the L ₂-norm solution is analogously given by

where . Both formulations in (26) and (27) can be efficiently solved as second order cone programming (SOCP) problems by a conic optimization solver (i.e., Sedumi 14 ) or as QCQP problems by a general QP solver (i.e., MOSEK 15 ). It is also known that a binary MKL problem can also be formulated as Semi-definite Programming (SDP), as proposed by Lanckriet et al. 4 and Kim et al. 16 . However, in a multi-class problem, SDP problems are computationally prohibitive due to the presence of PSD constraints and can only be solved approximately by relaxation 17 . On the contrary, the QCLP and QCQP formulations of binary classification problems can be easily extended to a multi-class setting using the one-versus-all (1vsA) coding, i.e., solving the problem of k classes as k number of binary problems. Therefore, the L _∞ multi-class SVM MKL is then formulated as

The L ₂ multi-class SVM MKL is given by

where

In LSSVM, the primal problem is given by 22

where most of the variables are defined in a similar way as in (24). The main difference is that the nonnegative slack variable ξ is replaced by a squared error term and the inequality constraints are modified as equality ones. Taking the conditions for optimality from the Lagrangian, eliminating , defining = [y ₁, ..., y _N ] and Y = diag(y ₁, ..., y _N ), one obtains the following linear system 22 :

where are unconstrained dual variables. Without the loss of generality, we denote and rewrite (37) as

In (38), we add an additional constraint as Y ^-2 = I then the coefficient becomes a static value in the multi-class case. In 1vsA coding, (37) requires to solve k number of linear problems whereas in (38), the coefficient matrix is only factorized once such that the solution of w.r.t. the multi-class label vectors is very efficient to obtain. The constraint Y ^-2 = I can be simply satisfied by assuming the class labels to be -1 and +1. Thus, from now on, we assume Y ^-2 = I in the following discussion.

To incorporate multiple kernels in LSSVM classification, the L _∞-norm approach is a QP problem, given by (assuming Y ^-2 = I)

The L ₂-norm approach is analogously formulated as

where . The λ parameter regularizes the squared error term in the primal objective in (36) and the quadratic term in the dual problem. Usually, the optimal λ needs to be selected empirically by cross-validation. In the kernel fusion of LSSVM, we can alternatively transform the effect of regularization as an identity kernel matrix in , where θ _{p + 1} = 1/λ. Then the MKL problem of combining p kernels is equivalent to combining p + 1 kernels where the last kernel is an identity matrix with the optimal coefficient corresponding to the λ value. This method has been mentioned by Lanckriet et al. to tackle the estimation of the regularization parameter in the soft margin SVM 4 . It has also been used by Ye et al. to jointly estimate the optimal kernel for discriminant analysis 17 . Saving the effort of validating λ may significantly reduce the model selection cost in complicated learning problems. By this transformation, the objective of LSSVM MKL becomes similar to that of SVM MKL with the main difference that the dual variables are unconstrained. Though (39) and (40) can in principle both be solved as QP problems by a conic solver or a QP solver, the efficiency of a linear solution of the LSSVM is lost. Fortunately, in a SIP formulation, the LSSVM MKL can be decomposed into iterations of the master problem of single kernel LSSVM learning, which is an unconstrained QP problem, and a coefficient optimization problem with very small scale.

As discussed, the dual L ₂ MKL solution can be extended to many machine learning problems. In principle, all MKL algorithms can be formulated in L _∞, L ₁, and L ₂ forms and lead to different solutions. To validate the proposed approach, we implemented and compared 20 algorithms on various data sets. The summary of all implemented algorithms is presented in Table 3. These algorithms combine L _∞ , L ₁, and L ₂ MKL with 1-SVM, SVM, and LSSVM. Moreover, to cope with imbalanced data in classification, we also extended Weighted SVM 23 24 and Weighted LSSVM 25 26 to their MKL formulations (presented in Additional file 1). Though we mainly focus on L _∞, L ₁, and L ₂ MKL methods, we also implement the L _n -norm MKL for 1-SVM, SVM, LS-SVM and Weighted SVM. These algorithms are applied on the four biomedical experimental data sets and the performance is reported in section 8 of Additional file 1. Moreover, the L _n -norm algorithms are also available on the website of this paper.

The performance of the proposed L ₂ MKL method was systematically evaluated and compared on six real benchmark data sets. The computational efficiency was compared on two UCI data sets. On each data set, we compared the L ₂ method with the L _∞, L ₁ and regularized L _∞ MKL method. In the regularized L _∞, we set the minimal boundary of kernel coefficients θ _min to 0.5, denoted as L _∞ (0.5). We also compared the three different optimization formulations SOCP, QCQP and SIP on the UCI data sets. The experiments were categorized in five groups as summarized in Table 4.

In the first experiment, the L ₂ 1-SVM MKL algorithm performed the best (Error 0.0780). As shown in Table 5, the L _∞ and L ₁ approaches all performed significantly worse than the L ₂ approach. For example, in the current experiment, when setting the minimal boundary of the kernel coefficients to 0.5, each data source was ensured to have a minimal contribution in integration, thereby improving the L _∞ performance from 0.0923 to 0.0806, although still lower than L ₂. In Figure 1 we illustrate the optimal kernel coefficients of different approaches. As shown, the L _∞ method assigned dominant coefficients to Text mining and Gene Ontology data, whereas other data sources were almost discarded from integration. In contrast, the L ₂ approach evenly distributed the coefficients over all data sources and thoroughly combined them in integration. When combining multiple kernels, sparse coefficients combine the model only with one or two kernels, making the combined model fragile with respect to the uncertainty and novelty. In real problems, the relevance of a new gene to a certain disease may not have been investigated thus a model solely based on Text and GO annotation is less reliable. L ₂ based integration evenly combines multiple genomic data sources. In this experiment, the L ₂ approach showed the same effect as the regularized L _∞ by setting some minimal boundaries on kernel coefficients. However, in the regularized L _∞, the minimal boundary θ _min usually is predefined according to the "rule of thumb". The main advantage of the L ₂ approach is that the θ _min values are determined automatically for different kernels and the performance is shown to be better with the manually selected values.

In the second experiment, recently discovered prostate cancer genes were prioritized using the same data sources and algorithms as in the first experiment. As shown in Table 6, the L ₂ method significantly outperformed other methods on prioritization of gene CDH23, and JAZF1. For 5 other genes (CPNE, EHBP1, MSMB, KLK3, IL16), the performance of the L ₂ method was comparable to the best result. In section 4 of Additional file 1, we also presented the optimal kernel coefficients and the prioritization results for individual sources. As shown in Additional file 1, the L _∞ algorithm assigned most of the coefficients to Text and Microarray data. Text data performs well in the prioritization of known disease genes, however, does not always work the best for newly discovered genes. This experiment demonstrates that when prioritizing novel prostate cancer relevant genes, the L ₂ MKL approach evenly optimized the kernel coefficients to combine heterogeneous genomic sources and its performance was significantly better than the L _∞ method. Moreover, we also compared the kernel based data fusion approach with the Endeavour gene prioritization software: for 6 genes the MKL approach performed significantly better than Endeavour.

One of the main contributions of this paper is that the L ₂ MKL notion can be applied on various machine learning problems. The first two experiments demonstrated a ranking problem using 1-SVM MKL to prioritize disease relevant genes. In the third experiment we optimized the L _∞, L ₁, and L ₂ -norm in SVM MKL and LSSVM MKL classifiers to support the diagnosis of patients according to their lymph node stage in rectal cancer development. The performance of the classifiers greatly depended on the selected features, therefore, for each classifier we compared 25 feature selection results (as a grid of 5 numbers of genes multiplied by 5 numbers of proteins). As shown in Table 7, the best performance was obtained with LSSVM L ₁ (error of AUC = 0.0325) using 25 genes and 15 proteins. The L ₂ LSSVM MKL classifier was also promising because its performance was comparable to the best result. In particular, for the two compared classifiers (LSSVM and SVM), the L ₁ and L ₂ approaches significantly outperformed the L _∞ approach. We also tried to regularize the kernel coefficients in L _∞ MKL using different θ _min values. Nine different θ _min were tried uniformly from 0.1 to 0.9 and the changes in performance is shown in Figure 2. As shown, increasing the θ _min value steadily improves the performance of LSSVM MKL and SVM MKL on the rectal cancer data sets. However, determining the optimal θ _min was a non-trivial issue. When θ _min was smaller than 0.6, the performance of LSSVM MKL L _∞ remained unchanged, meaning that the "rule of thumb" value 0.5 used in experiment 1 is not valid here. In comparison, when using the L ₂ based MKL classifiers, there is no need to specify θ _min and the performance is still comparable to the best performance obtained with regularized L _∞ MKL.

In LSSVM kernel fusion, we estimated the λ jointly as a coefficient assigned to an identity matrix. Since the number of samples is small in this experiment, the standard cross-validation approach to select the optimal λ on validation data was not tried. To investigate whether the estimated λ value is optimal, we set λ to 51 different values uniformly sampled on the log ₂ scale from -10 to 40. We compared the joint estimation result with the optimal classification performance among the sampled λ values. The joint estimation results were found as optimal for most of the results. An example is illustrated in Figure 3 for the integration of four kernels constructed by 27 gene features and 17 protein features. The coefficients estimated by the L _∞-norm were almost 0 thus the λ values were very big. In contrast, the λ values estimated by the non-sparse L ₂ method were at reasonable scales.

In the fourth experiment we validated the proposed approach on three clinical data sets containing more samples. On the endometrial and miscarriage data sets, we compared eight MKL algorithms with various norms. For the imbalanced pregnancy data set, we applied eight weighted MKL algorithms. The results are shown in Table 8, 9, and 10. On endometrial data, the difference of performance was rather small. Though the two L ₂ methods were not optimal, they were comparable to the best result. On miscarriage data, the L ₂ methods performed significantly better than comparing algorithms. On pregnancy data, the weighted L ₂ LSSVM MKL and weighted L ₁ LSSVM MKL performed significantly better than others. We also regularized the kernel coefficients using different θ _min values on LSSVM L _∞ and SVM L _∞ MKL classifiers. The results are presented in Figure 4, Figure 5 and Figure 6. As shown, the optimal θ _min value differs across data sets thus the "rule of thumb" value of 0.5 may not work for all the problems. For the endometrial and miscarriage data sets, the optimal θ _min for both MKL classifiers is 0.2. For pregnancy data set, the optimal θ _min value for LSSVM is 1 and for SVM 0.9. In comparison, on the miscarriage and pregnancy data set, the performance of the L ₂ algorithm is comparable or even much better than the best regularized L _∞ algorithm. For the endometrial data set, though the optimal regularized L _∞ LSSVM and SVM MKL classifiers outperform L ₂ classifiers, L ₂ methods still perform better than or as equal as the unregularized L _∞ method.

To investigate whether the combination of multiple kernels performs as well as the best individual kernel, we evaluated the performance of all the individual kernels in section 5 of Additional file 1. As shown, the clinical kernel proposed by Daemen and De Moor 33 has better quality than linear, RBF and polynomial kernels on endometrial and pregnancy data sets. For the miscarriage data set, the first RBF kernel has better quality than the other seven kernels. Despite the difference in individual kernels, the performance of MKL is comparable to the best individual kernel, demonstrating that MKL is also useful to combine candidate kernels derived from a single data set.

The effectiveness of MKL can also be justified by investigating the kernel coefficients optimized on all the data sets and classifiers. As shown in section 6 of Additional file 1, the kernel coefficients optimized by L _∞ MKL algorithms were sparse whereas the L ₂ ones were more evenly assigned to different kernels. The best individual kernels of all data sets usually get dominant coefficient, explaining why the performance of MKL algorithms is comparable to the best individual kernels.

In this paper, the regularization parameter λ in LSSVM classifiers was jointly estimated in MKL. Since the clinical data sets contain a sufficient number of samples to select the λ by cross validation, we systematically compared the estimation approach with the standard validation approach to determine the λ values. As shown in Table 11, the estimation approach based on L _∞ performed worse than the validation approach. This is probably because the estimated λ values are either very big or very small when the kernel coefficients were sparse. In contrast, the L ₂ based estimation approach yielded comparable performance as the validation approach. We also benchmarked the performance of LSSVM MKL classifiers using 21 different static λ values on the data sets and the results are shown in section 7 of Additional file 1. In real problems, to select the optimal λ value in LSSVM is a non-trivial issue and it is often optimized as a hyper-parameter on validation data. The main advantage of L ₂ MKL is that the estimation approach is more computational efficient than cross validation and yields a comparable performance.