In machine learning, the receiver operating characteristic (ROC) curve is used to evaluate the discriminative performance of binary classifiers. This is obtained by plotting the curve of the true positive rate (Sensitivity) versus the false positive rate (1 – Specificity) for a binary classifier by varying the discrimination threshold.

All the calculations of true positive rate and false positive rate are attained when using a particular classifier threshold. By varying the threshold, a set of values for these measurements is obtained. This set of values is plotted in a two-dimensional Cartesian graph to yield the ROC curve. The ROC curve takes into account all the possible solutions by varying the discriminative threshold. The best performance would be produced, if the ROC curve matches with the upper left corner of the ROC space (which yields 100% sensitivity and 100% specificity). The closer the ROC curve is to the upper part of the ROC space, the better the performance of the classifier.

An ROC curve is a two dimensional illustration of classifier performance. Reducing ROC performance to a single scalar value to represent expected performance helps compare classifiers. A popular method is to calculate the area under the ROC curve (AUC) 6.

The AUC, being a part of the area of the unit square, has a value between 0 and 1. Since random guessing could produce the diagonal line between (0, 0) and (1, 1) with an area of 0.5, a classifier with an AUC less than 0.5 is undesirable 7. An AUC value close to 1 indicates better performance for a binary classifier 8.

We rank all genes of the training set using the first step of Mamitsuka 9's ROC, which is the equivalent of the Mann-Whitney U statistic 10, normalized by the number of possible pairings of positive and negative values, also known as the two sample Wilcoxon rank sum statistic 6. The AUC actually represents the probability that a randomly chosen positive example is correctly rated (ranked) with greater suspicion than a randomly chosen negative example.

Let us consider a training dataset D MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaWenfgDOvwBHrxAJfwnHbqeg0uy0HwzTfgDPnwy1aaceaGae83aXteaaa@374F@ of n examples, where each example comprises m attributes: x₁, x₂, x₃,..., x_m. Each of the m attributes has a differing discriminative power reflected by its respective AUC. To calculate the discriminative power that is expressed in terms of AUC, we plot the ROC curve for each gene paired with the class label, (i.e., {x_i, Y_i}, where 1 ≤ i ≤ m and Y is the vector of class labels) and calculate the AUC of the ROC curve. Now, we order the genes based on their respective AUCs.

We attempt to classify the validation dataset with the top ranked genes. At first we pass the top 10 genes individually to all classifiers, and note the classification accuracy of each classifier on a validation set. For the second pass, we select the gene for which the most classifiers achieve their highest accuracy. Then we form a pair of the selected gene from the first pass, along with the remaining nine genes, and input these nine pairs to the selected classifiers. We then note the classification accuracy. The pair on which the most classifiers achieve their highest accuracy is selected and given to a third pass. We continue adding single genes to form 3-gene combinations, and so on.

A diverse set of fifteen classifiers was used for this process. They are: Logistic Model Tree (LMT) 11, Naïve Bayes Tree (NBTree) 12, Naïve Bayes, Random Forest, C4.5, k-Nearest Neighbour (k-NN), Artificial Neural Network (ANN), Logistic Regression, Support Vector Machine (SVM), and bagging 13 and boosting (ADABoost.M1) 14 for Naïve Bayes, Random Forest and C4.5.

We stop growing the gene set once more than 50% of the classifiers have their accuracy lowered on the validation set by the addition of an individual gene. In the case of any tie between two or more genes, we note the total accuracy (out of 1500%) for the tied genes and the gene with the largest total accuracy is chosen for the next pass. This majority voting approach allows us to select a small subset of genes that can boost classification accuracy on a number of classifiers. One can then tune the performance of an individual classifier by choosing the prefix of the genes (ranked using the voting approach) that delivers best accuracy. For example, for C4.5 with boosting on the van 't Veer data, adding the 5th gene to the first four genes actually degrades the performance of that specific classifier. So for that individual classifier, rather than using our final selection of 7 genes, we can instead use only a subset of 4 (out of the 7) genes.

In each of the three datasets used in our analysis, the prognostic outcome to be predicted is whether distant metastases will occur within 5 years (poor prognosis) or whether the patient is disease-free after 5 years (good prognosis).

van't Veer data 3: The dataset comprises 97 breast cancer patients treated through modified radical mastectomy or breast-conserving treatment followed by radiotherapy. The patients were split into a training set of 68, a validation set of 10, and a test set of 19 cases. The training set consists of 29 positive (poor prognoses) and 39 negative (good prognoses) cases, the validation set comprises five positive and five negative cases, and the test set was made up of 12 positive and 7 negative cases. Further, we created a merged dataset from van 't Veer's 3 training (our training and validation set) and test sets to apply k-fold cross validation (CV). In k-fold CV, 10 cases out of the training set are randomly selected for the validation set before applying the FROC.

Ma et al. data 4: This dataset contains 60 breast cancer patients treated through standard breast surgery followed by continued adjuvent tamoxifen therapy. There were 28 positive cases (poor prognoses) and 32 negative cases (good prognoses). We separate the first 5 positive cases and the last 5 negative cases to form the test set and use the remaining cases for training.

Loi et al. data 5: The dataset is made up of 77 breast cancer patients obtained from the GUYT2 test data used in the Loi et al. study, with similar treatments to those performed in the Ma et al. dataset. There were 10 positive cases (poor prognoses) and 67 negative cases (good prognoses). This dataset was included in our study as a completely independent test dataset. All patients were considered as test cases to gauge performance of the classifiers trained on the Ma et al. dataset.

For the van 't Veer 3 and Ma et al. 4 datasets, we used a holdout cross validation procedure, where a training set is used to train the classifiers and a separate test set is used to evaluate. We have also used the more general k-fold cross validation (CV) scheme to evaluate the performance on van 't Veer data. In k-fold CV, the original sample is partitioned into k subsamples. Of the k subsamples, a single subsample is retained as the validation data for testing the model, and the remaining k - 1 subsamples are used as training data. The CV process is then repeated k times (the folds), with each of the k subsamples used exactly once as the validation data. The k results from the folds, then, can be averaged (or otherwise combined) to produce a single estimation. Here, we have chosen k to be 5. The results for k-fold CV is presented in the Additional File 1.

All of our evaluation results are reported in weighted accuracy 15, which is calculated by the formula shown in Eq. (1).

Weighted Accuracy = T P P + T N N 2 MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaee4vaCLaeeyzauMaeeyAaKMaee4zaCMaeeiAaGMaeeiDaqNaeeyzauMaeeizaqMaeeiiaaIaeeyqaeKaee4yamMaee4yamMaeeyDauNaeeOCaiNaeeyyaeMaee4yamMaeeyEaKNaeyypa0tcfa4aaSaaaeaadaWcaaqaaiabdsfaujabdcfaqbqaaiabdcfaqbaacqGHRaWkdaWcaaqaaiabdsfaujabd6eaobqaaiabd6eaobaaaeaacqaIYaGmaaaaaa@4CA8@

where, P = Total number of positive cases,

N = Total number of negative cases.

As the treatment procedures applied to the patients in both the van 't Veer study 3 (no adjuvent therapy) and Ma et al. study 4 (adjuvent therapy with tamoxifen) are vastly different, it is not surprising that there is no overlap between the two gene sets identified as the best predictors of prognosis outcome. The biology driving the chance to distant metastasis in each dataset is likely to be significantly different and as such it would not make sense to expect the gene lists to overlap. Therefore, we will consider each gene set independently.

Gene Ontology (GO) and Kyoto Encyclopedia of Genes and Genomes (KEGG) pathway analysis of the two gene sets show that each of the genes are diverse in function and appear to be unrelated in terms of the biological processes in which they are involved. What is interesting, however, is that the majority of the genes have been previously shown to be related to cancer in the literature (as shown below). This suggests that our feature selection procedure yields a compact sampling of the diverse biological processes represented by the microarray, which are highly representative of the prognostic potential of the patient. In concordance with this, in the 7 gene set, TSPYL5 and CA9 have been previously used as prognostic biomarkers in cancer 1516171819. Furthermore, four of the top 7 genes selected by our method are in the set of 231 genes used in the study by van 't Veer 3. and the most important individual gene in improving a number of the classifier performances in the test set (TSPYL5) is present in the 17 genes selected by Alexe et al. 15. In the 6 gene set, CD55 has been used previously as a prognostic biomarker in gastric cancer 20.

A number of efforts have been made in this direction for breast cancer prognosis but without major success. Ritz 40 combined both genetic and clinical information in a neural network for breast cancer prognosis, but found that the combination did not improve the performance.

Dettling et al. 41 applied penalized logistic regression analysis to predict cancer prognosis for the van 't Veer 3 dataset. They found that none of the clinical variables entered the model and concluded that the clinical data did not contain any useful independent information for prediction, given the gene expression profile.

To prognosticate on the breast cancer dataset, Alexe et al. 15 applied the Logical Analysis of Data (LAD) tool to analyze microarray data. They identified 17 genes out of 25,000 possible genes that could distinguish patients with Poor or Good prognoses. Amongst the 17 genes the LAD tool identified three and five genes that were associated with Poor and Good prognoses, respectively. Two wholly new classes (defined by similar sets of covering patterns, gene expression ranges, and clinical features) of patients were discovered. It was also demonstrated that the training and test sets of van 't Veer 3 differ in their characteristics. However, this study of Alexe et al. was overly specific to the chosen classifier (the LAD tool), as we shall shortly see.

We assessed the classification performance of five different subsets of genes on the van 't Veer 3 dataset. The 231 and 70 genes selected by van 't Veer 3, the set of 17 genes selected by the LAD technique 15, a set of 17 genes selected by ROC (FROC) with Markov blanket 9 and the set of 7 genes selected by our voting approach. For ROC (FROC) with Markov blanket we have used the same parameter for the number of genes to select as given in that paper, namely 50 genes and then using an area between two ROC curves (ABR) value > 20 for the second step, chosen to yield the most competitive performance for the technique. We applied five classification methods used by Alexe et al. 15 and our top performing classifier C4.5 with boosting on the gene set of size 4. These classification methods include ANN, SVM, Logistic Regression, k-NN, C4.5 decision tree and C4.5 with boosting (see Table 6). Following this, predictive models were constructed for the training set and were tested using the supplied test set of 19 samples.

It is clear that the weighted accuracy in distinguishing patients with Good and Poor breast cancer prognoses is the highest across all classifiers using the 7 genes selected by our voting approach and is much higher than the models using 17 (by LAD), 70 and 231 genes. Our approach produced much better performance for most classifiers, except for ANN, where using the 17 selected genes of Alexe et al. 15 or by using ROC with Markov blanket 9 was better. However, the methodology of Alexe et al. incorporated a "selection bias" 42 for finding their subset of 17 genes, since the test set was used. In contrast, our voting approach did not have access to the test set for gene selection. When selecting genes using our voting approach, we used only the training dataset, keeping the test set completely unknown. The performance of the classifiers using the selected 70 and 231 genes by van 't Veer 3 was found to be insignificant compared with that of our approach. Furthermore, 70 or 231 is not a compact set of genes and our voting method can obtain a better accuracy using at most four genes (see Table 2).

Gevaert et al. 43 proposed a Bayesian networ k-based strategy to treat clinical and microarray data along the same lines as above using the same dataset. A probabilistic model was used because it integrates the data sources in several ways, and explores and documents the model structure and parameters. The concept of a Markov Blanket is used to identify all the variables that shield the class variable from being affected by the rest of the network. However, all the processes are integrated in the classifier, and hence the performance of the system is biased towards the choice of a classifier. Furthermore, the performance of the classifier would depend on the selection of the initial distribution for the model.

Jeffery et al. 44 have demonstrated that the ROC is an accurate way to identify differentially regulated genes in a microarray dataset and that it can produce robust classifiers applying 9 feature selection techniques on 9 gene expression datasets. When dealing with datasets that have 15 or more samples, the ROC was shown to be the most accurate. Other filter approaches like t-test and Principle Component Analysis (PCA) produce reasonable results, but ROC yields better results (see Figure 1) on the van 't Veer 3 dataset. It is particularly useful for gene expression data, as it is not directly dependent on the standard deviation of the expression of each gene like the t-test is, or only on the correlation of each genes like the PCA is. Moreover, unlike PCA, ROC is not sensitive to the scaling of the data.

It is interesting to consider the results of gene set enrichment analysis (GSEA) 45 on our set of 7 genes obtained by voting, against the other gene sets proposed by van 't Veer et al. 3 (size 70 and 231 genes, respectively), Alexe et al. 15 (17 genes), and the one we have obtained using Mamitsuka's 9 technique (17 genes). Three out of the five gene sets are enriched in phenotype 1 (i.e. relapse). Of which, Mamitsuka's and our gene sets have an FDR q-value equal to 0 and 0.005, with an enrichment score (ES) of 0.78 and 0.80, respectively. Members of the leading edge subset (i.e., tags = 100%, list = 20% and signal = 125%) also indicate that our gene set contains only those genes contributing to the enrichment score, compared to the other gene sets that contain only a fraction of genes contributing to the enrichment score (see Additional File 2).