To consider the differentiating power of a set of markers consisting of k markers, we first define the score of the marker set. A normal sample contains normal proto-oncogenes that promote cell growth and mitosis and tumor suppressor genes that discourage cell growth. During cancer development, proto-oncogenes can be mutated by carcinogenic agents to become oncogenes, which produce excessive levels of growth promoting proteins. Cancer results from cumulative mutations of proto-oncogenes and suppressor genes which together allow the unregulated growth of cells. Hence, cancer development involves uncontrolled cell division resulting from a series (progression) of gene mutations that typically involve two categories of function: promotion of cell division and inactivation of cell cycle suppression. The expression values of excessive growth genes tend to be much higher than the same genes of a normal sample. Similarly, the level of tumor suppressor genes of a cancer sample tends to be much lower than a normal sample. It is relatively rare that a single marker alone could offer sufficient power to differentiate all the classes well in multi-class data. So we consider marker sets with at least two markers keeping in mind that over-growth of some genes and inactivation of other genes often happen together in cancer cases. Later stages of cancer involve tissue invasiveness, during which malignant cells travel among tissues via the circulatory and/or lymphatic system and grow and thrive in their new locations. Therefore the relative amount of two or multiple markers in a sample could be an indication of the cancer stages.

For markers i and j, we use the following notation. Let f _1mij , m = 1,... M, represent the frequency count of samples in class C_m that satisfy the condition: the expression value for marker i is less than the expression value of marker j. Similarly, let f _2mij , m = 1,... M, be the frequency count of samples that satisfy the condition: the expression value for marker i is greater than or equal to the expression value of marker j. These counts can be presented in a cross-tabulation table as shown in Table 1, where f _1mij , m = 1,... M, are the entries in the first column and f _2mij , m = 1,... M, are the entries in the second column.

Based on the cancer mechanism that there is excessive growth in tumor cells and inactivation of suppressor genes, the best informative genes would consist of some genes overly expressed and some other genes that are down-regulated. In particular, a marker pair with genes i and j become increasingly more informative of the cancer status as the difference of their expression values diverges away from the corresponding difference between the same marker pairs of a normal patient. Consequently, for two markers encoding genes or proteins that are important to differentiate the cancer status, their relative magnitude of the expressions are inter-related and whether the expression value for marker i is less than the expression value of marker j is not independent of the class status. To incorporate the sample size information, the Chisquare statistic defined in equation (1) can be used to assess whether the pair of markers i and j are informative for classification of cancer status:

χ i j 2 = Σ q = 1 2 Σ m = 1 M ( f q m i j - n m T q / N ) 2 n m T q / N = N Σ q = 1 2 Σ m = 1 M f q m i j 2 n m T q - 1 ,

where n_m and T_q are the row and column totals from the m^th row and q^th column, respectively. If all the counts in Table 1 are large and all cell counts are at least five, a traditional way to declare significance for the pair is to compare the calculated statistic with the chi-squared-distribution with M -1 degrees of freedom. However, the significance level for declaring significance of a single test needs to be adjusted for multiple comparisons. There are various directions including family-wise error rate control, false-discovery rate (FDR) control, among others. The family-wise error rate control tends to be conservative while the FDR control could lead to high false positive rates. In this work, we do not use the chi-squared-distribution and do not decide how many pairs are significant. Instead, we only use the Chi-square statistic in (1) as an indication of how much departure from independence between the class and the chance of observing marker i expression value less than that of marker j. As the departure from independence increases, the chi-squared statistic value increases. We select the top pair of markers that yield the highest value of the chi-squared statistic. Additional marker selection will follow the algorithm in section 2.2.

For a set that contains k markers, we consider the cross tabulated Table 2 that contains frequency counts of all unique pairwise comparisons among the k markers. There are k(k-1) columns of counts. The sum of counts in each column remains to be the same as that for two-marker case. The row totals are now k times that of the sample sizes in corresponding classes. We calculate the Chisqure statistic as in equation (2).

χ 2 i 1 . . . i k = Σ a = 1 k - 1 Σ b = a + 1 k Σ m = 1 M Σ q = 1 2 f q m i a i b - k n m T q N 2 k n m T q / N = N Σ a = 1 k - 1 Σ b = a + 1 k Σ m = 1 M Σ q = 1 2 f q m i a i b 2 k n m T q - 1 ,

Note that the χ 2 i 1 . . . i k only differs from Σ a = 1 k - 1 Σ b = a + 1 k χ 2 i a i b by the division factor k in the first term. So comparison of χ 2 i 1 . . . i k and χ 2 j 1 . . . j k for two sets of k-markers {i₁,..., i_k } and {j₁,..., j_k } is equivalent to comparing Σ a = 1 k - 1 Σ b = a + 1 k χ 2 i a i b and Σ a = 1 k - 1 Σ b = a + 1 k χ 2 j a j b . The latter can be calculated easily without much computational cost after the statistics for marker pairs have been computed. The statistics given in equation (2) should be restricted to comparing marker sets with the same number of markers. All the k-marker sets can be ranked according to the magnitude of the Chisquare statistic values. The k-marker set with the highest Chisquare value is the most informative set among all k-marker sets.

For comparing multiple sets with different numbers of markers, the Chisquare statistics given earlier can not be used because they accumulate different numbers of terms. In such case, we use the leave-one-out cross validation (LOOCV) accuracy within the training data obtained with the procedure below as the objective function.

Suppose the training data contains N_tr samples. Without loss of generality, we use Ω_tr = {S₁ ,..., S_Ntr } to denote the collection of these samples. For a marker set {i₁,..., i_k }, the LOOCV is performed within this training sample. In particular, we

1. Leave out one training sample S_l to be used as the test data and use the rest of the training samples Ω_tr\ S_l as training data.

2. For class m, (1 ≤ m ≤ M), assign S_l to this class and calculate the Chisquare statistic for the marker set {i₁,..., i_k }. We obtain M Chisquare statistics { χ 2 ( m ) i 1 . . . i k , m = 1,..., M}. The predicted class m ^ ( S l ) for S_l is the class that has the maximum Chisquare statistics, i.e., m ^ ( S l ) = arg max 1 ≤ m ≤ M χ 2 ( m ) i 1 . . . i k .

3. Repeat 1 and 2 for all the samples in Ω_tr to get the prediction of all samples in Ω_tr: m ^ ( S l ) , l = 1,..., N_tr .

4. The LOOCV accuracy for marker set {i₁,..., i_k } is LOOCV(i₁,..., i_k ) = the proportion of correctly classified samples in Ω_tr.

The comparison of marker sets of different sizes can be based on the LOOCV accuracy within the training data. The marker set with highest LOOCV is more informative than other marker sets. For marker sets with the same number of markers and identical Chisquare statistic value, we also use the LOOCV accuracy as a measure of their differentiating power toward the different cancer classes.

For a given upper bound B on the cardinality of the marker set, the marker selection process first selects the top scoring pairs and then sequentially adds additional markers into the active set until the total number of markers in the active set reaches the upper bound B. This is done following the algorithm below: Denote the set of remaining markers as ẞ. The initial value of ẞ is the list of all markers.

1. Calculate and record the Chisquare statistics for all marker pairs using the training data.

2. If the highest value of the Chisquare statistics is achieved by a unique marker pair, select this pair and denote it as TS₂. Calculate the LOOCV for this pair of markers and denote it as LOOCV₂. Update the remaining marker set ẞ by removing the marker pair selected.

3. If there are multiple marker pairs that have identical maximum Chisquare statistic value, calculate the LOOCV accuracy of these marker pairs using the training data. Keep the marker pairs that have the highest LOOCV accuracy. If the highest accuracy is achieved by more than one pairs, denote the different pairs as TS_2,1, TS_2,2, etc.

4. Find the top scoring triplets by adding additional marker to the top scoring pairs. This is done as follows. For each of the top scoring pairs resulting from 2 and 3, find the marker from the list of remaining markers ẞ such that the triplet has the highest Chisquare statistic value. If there are multiple triplets with identical maximum Chisquare value, calculate the LOOCV accuracy of these triplets and record those triplets that yield the highest LOOCV accuracy. Denote the top scoring triplet as TS₃ if it is unique, and TS_3,1, TS_3,2, etc otherwise if multiple sets achieved identical accuracy. Record the LOOCV accuracy of the top scoring triplets.

5. For k = 4, 5,..., B, find TS_k and their corresponding LOOCV accuracy LOOCV_k. As k increases, the set TS_k tend to be unique.

6. Select the smallest k-marker set such that the LOOCV is maximized over all TS_k, k = 1,..., B. If the marker set is not unique, randomly select one of them as the final set. Denote the final selected informative marker set as TS_G, where G = arg max_{1 ≤ k ≤ B} LOOCV_k.

As discussed in section 2.1.2, the comparison of the Chisquare statistics for χ 2 i 1 . . . i k can be simplified by comparing the summation of all Chisquare statistics from unique marker pairs Σ a = 1 k - 1 Σ b = a + 1 k χ 2 i a i b .

As an illustration, Figure 1 shows the the accuracy of the training and test data for k = 2,..., 100 for the 6-class DLBCL cancer microarray data 17 . It can be seen that the training accuracy reached maximum when k = 16. So the selected marker set is TS₁₆.

Our experience suggests that it is sufficient to set the upper bound B to be 50 if the total number of classes M ≤ 4, 100 if 5 ≤ M ≤ 8, and 150 if M ≥ 9.

To predict the class information for each sample in the test data, we use the selected marker set and calculate the scores of this sample belonging to each class. A large value for a class suggests that putting this sample in that class helps to increase the separation of different classes. The predicted class is set to be the one that has the largest score. In particular, suppose the selected marker set consists of markers m₁, m₂,..., m_k , the training data is Ω_tr, and the sample to be predicted is x _new. Let χ 2 i 1 . . . i k | C i be the Chisquare statistic value when we put the sample in class C_i, i = 1,..., M. There are M Chisquare values. We assign the sample to the class with the largest Chisquare value:

Class of x new = arg max i = 1 , . . . , M χ 2 i 1 . . . i k | C i .

If multiple classes reach the same maximum Chisquare value, we further calculate the LOOCV accuracy for these classes. The final prediction is based on which class achieves the highest LOOCV accuracy.

The performance of the proposed TSG marker selection and classifier is evaluated on both binary and multi-class microarray expression data. We consider the 19 datasets that were used for evaluation of TSP, k-TSP and their multi-class version classifiers in Tan et al. 2005. There are 9 binary and 10 multi-class datasets. These datasets are related to human cancers including colon cancer, leukemia, central nervous system, diffuse large B-cell lymphoma, breast cancer, lung cancer, and prostate cancer. The reference, sample size, number of genes in each dataset, and the number of samples in each class are summarized in Tables 3 and 4. The number of classes ranges from 2 to 14. The number of markers ranges from 2000 to 16063. Average number of samples per class ranges from 13 to 140. The ratio between the number of samples per class and the number of markers ranges from 0.000845 to 0.0155.

First, we consider comparison of TSG and k-TSP classifiers for binary datasets based on 5-fold cross validation. The subjects in each class are randomly partitioned into 5 parts, 4 of which form the training data and the rest of the subjects constitute the test data. The feature selection and modeling were conducted on the training data and prediction for each subject in the test data was given. For the results to be comparable to the TSP family classifiers, we also follow the same comparison methods as in Tan et al. 3 . In particular, we perform LOOCV for binary datasets and perform independent test for multi-class datasets. In the LOOCV, each sample is taken out and the remaining N-1 samples are used to train the classifier, which is then used to predict the class label of the leave-out sample. The LOOCV accuracy is the proportion of correctly classified samples. Each of the multi-class datasets is partitioned into training and test data. We follow exactly the same partition scheme as in Tan et al. 3 .

Since an objective of this research is to improve the TSP family classifiers, we present the percent of increase in classification accuracy in barplots. The percent of increase for TSG over TSP is defined as (accuracy of TSG - accuracy of TSP)/accuracy of TSP x 100%.

The percent of increase for any two classifiers are similarly defined. As TSP classifier uses two genes, k-TSP and TSG use at least two genes, we are particularly interested in comparing the improvement in accuracy for TSG over TSP and k-TSP over TSP in binary classifications. Similarly, in the multi-class cases, we are interested in comparing the increase in accuracy for TSG over HC-TSP and HC-k-TSP over HC-TSP.

For reference, we also include the classification accuracy of decision trees (DT), Naive Bayes (NB), k-nearest neighbor (k-NN), Support Vector Machines (SVM) and prediction analysis of microarrays (PAM) in our comparison tables when they are available from the literature. These results were reported in Tan et al. 3 for leave-one-out cross validation for binary data and independent test for multiclass data. We include them only for convenience of discussion. DT and PAM have feature selection function while NB, k-NN and SVM perform classification using the entire set of features. Since DT, k-NN, and NB in general have lower accuracy than the other classifiers, we focus our discussion on other classifiers.

In this section we present the comparison of the TSG marker selection algorithm with other algorithms using 9 benchmark binary class cancer expression datasets. These data have been analyzed extensively by many authors with wrapper, filtering and ensemble methods.

For the multi-class datasets, the accuracy of classifiers on independent test data is presented in Table 7.

The percent of increase in accuracy for HC-k-TSP and TSG over HC-TSP is shown in Figure 4. The similar bar heights for the first and third bars in Leuk1, Leuk2, SRBCT, and GCM datasets indicate that HC-k-TSP and TSG improve upon TSP with similar amount for these datasets. For the Breast, DLBCL, Leuk3 datasets, TSG achieved a lot more accuracy gain over HC-TSP than HC-k-TSP. For these three datasets, the gain of accuracy of TSG over HC-k-TSP is 30%, 12%, and 10.87% respectively. For Lung1, Lung2 and on average, TSG also has better accuracy than HC-k-TSP but the gain of TSG over HC-k-TSP is not more than 5%. For the cancers dataset, TSG lost 3.27% accuracy than HC-k-TSP. In summary, TSG gains more accuracy than HC-k-TSP in all except for one dataset. We remark that there are three schemes of extending binary classifier TSP to multi-class classifiers (1-vs-1, 1-vs-others, and hierarchical classification schemes). The HC-k-TSP and HC-TSP results reported in Tan et al. 3 are with the hierarchical scheme that performs best out of all three schemes. Therefore, TSG classifier has even better performance than the 1-vs-1 and 1-vs-others multi-class extensions of TSP family classifiers.

The accuracy based on independent test samples for TSG, HC-k-TSP, and SVM relative to the accuracy of PAM is presented in the right panel of Figure 3. For half of the datasets SVM has better performance than PAM and for the other half of the datasets, PAM performs better than SVM in accuracy. So on average, SVM and PAM are comparable. The accuracy of HC-k-TSP in majority of datasets is below that of PAM. On average, HC-k-TSP has accuracy slightly lower than PAM. TSG has better accuracy than PAM in five of ten datasets, and TSG is not as accurate as PAM in four datasets. On average, TSG, PAM and SVM have comparable performance with TSG only slightly better. In summary, for multi-class problems, TSG, SVM and PAM are better than HC-k-TSP in accuracy.

For comparison of classifiers with finite number of samples, the number of genes used by each classifier is an important factor. Classifiers using more genes tend to overfit the data. Hence classifiers with small number of genes are preferred. Since SVM, k-NN, and NB use the entire set of genes in their classification algorithm, we eliminate them from further discuss on number of genes used and restrict the rest of the discussion in this subsection to TSP, HC-TSP, DT, k-TSP, HC-k-TSP, TSG, and PAM. For these classifiers, we plot the number of genes used in each dataset for each classifier. For quite many datasets, PAM used hundreds or thousands of genes in the final classifiers. So we set the upper limit of the vertical axis in Figure 5 to be 50 in binary cases and 140 in multi-class cases so that the numbers used by other classifiers can be seen. The numbers for the same classifier under different datasets are connected for convenience of viewing.

It can be seen from Figure 5 that TSP classifier has the lowest number of genes in binary data classification. PAM used much more number of genes than other classifiers. The classifier that has the second smallest number of genes in binary data classification is TSG, followed by DT that uses the third smallest number of genes. The k-TSP classifier in general used more number of genes than TSP, TSG, and DT.

For multi-class data, DT uses the least number of genes followed by HC-TSP in the second place. However, as discussed in Section 3.1, DT and HC-TSP are not as accurate as SVM, PAM, HC-k-TSP, and TSG. In 4 out of 10 datasets TSG used more genes than HC-k-TSP and in 5 datasets TSG used fewer genes than HC-k-TSP. On average across all 10 datasets, TSG uses 51 genes and HC-k-TSP uses 53.4 genes. PAM consistently uses a lot more genes than other classifiers except in Lung1 data. In fact, the number of genes used by PAM is in the magnitude of thousands in order to reach comparable accuracy as TSG. Recall that on average TSG, PAM and SVM have comparable accuracy for independent test data and HC-k-TSP has lower performance. Now combining the accuracy and the number of genes used, TSG outperforms the rest in that it uses smaller number of genes to reach high accuracy. Smaller number of genes makes it feasible to perform follow-up studies and further experimental verification after the informative genes selection.

The TSG classifier has an easy interpretation. Recall that the main rule to classify a sample is based on which class gives the highest Chisquare statistic value when the sample is assigned to that class (see section 2.3). Note that there is a one to one correspondence between the Chisquare statistic values and the degree of departure from independence between the class label and the column variables in Table 2. The bigger the Chisquare statistic, the further the departure is from independence and vice versa. So the class label assignment tries to maximize the dependence between the classes and the selected variables for a given set of training data and a test sample.

The TSG classifier has the same interpretation as the TSP classifier. In particular, suppose the expression values for these two genes are E₁ and E₂ for a sample, the prediction of the class label of this sample depends on whether E₁ < E₂. For Leuk, CNS, and pros3 data, TSG achieved accuracy 98.61, 97.06, and 100 for these three datasets respectively using 2 genes. Also using 2 genes, TSP only achieved 93.80, 77.90, 97.00 in LOOCV accuracy for these three datasets. Therefore, TSG finds genes that are even more informative than TSP classifier in these three datasets. When the selected informative genes have more than 2 genes, then the class prediction of a sample depends on all pairwise comparisons of the expression values E i 1 , ⋯ , E i k for the selected genes i₁, i₂,..., i_k from this sample, i.e., which of the inequalities E i 1 < E i 2 , E i 1 ≥ E i 2 ,......, E i k - 1 < E i k , E i k - 1 ≥ E i k are true. Due to the similar interpretation to the TSP family classifiers, we do not reiterate for TSG and refer the readers to Tan et al. 3 for details.