In this study, we used two microarray data sets: one from the Botany Array Resource at the University of Toronto 22, and the other from the AtGenExpress Consortium 23, archived at NASCArrays 2425. These data sets include over 1000 expression-level experiments for Arabidopsis, and using all of them would give a data set with dimensionality over 1000. Since the performance of statistical and machine-learning methods tends to decrease with dimensionality, we chose only those experiments that are specifically stress-related. Even so, the covariance matrix of the resulting data set is singular, which is a problem for many of the machine-learning methods. The singularity is probably due to dependencies between the expression levels under control conditions, since removing the controls from the data sets solved the problem. To compensate, we tried applying the learning algorithms to expression-level ratios (i.e., ratios of experimental to control conditions). However, we found that the results were better when ratios were not used (data not shown). This is probably because the classifiers look for genes that respond similarly to the known stress-associated genes, so it is not so important to include the controls. In addition, since many of the features are time-courses, there is still a "time zero" control included for the values, providing a baseline measurement. The results reported in this article are therefore based on absolute expression levels without controls.

From the Toronto data set, we selected 54 features corresponding to experiments conducted primarily to study plant environmental and stress physiology, plant physiology, plant-microbe and plant-insect interactions. From the AtGenExpress data set, we selected 236 features, including various abiotic stresses (e.g., osmotic stress, heat stress, cold stress, salt stress, drought stress, UV-B stress, wounding stress, water-deprivation stress and oxidative stress). We combined the selected features into a single data set. The resulting data set consists of gene expression levels for 22,746 genes under 54 + 236 = 290 different experimental conditions.

We used terms from the Gene Ontology for Biological Processes (GOBP) to represent gene function. For example, the GOBP term GO:0006950 [response to stress] refers to genes that respond to stress. In general, the Gene Ontology (GO) provides a dynamic controlled vocabulary for describing genes and gene products in any organism 26. "Biological Process" is one of three broad GO categories (the other two being "Molecular Function" and "Cellular Component"). GOBP terms are organized into a directed acyclic graph (DAG) to reflect the hierarchical relationships between the terms. Parent GOBP terms are subdivided into increasingly specific child GOBP terms.

Since our study focussed on stress, we were concerned with gene functions at or below the term GO:0006950 [response to stress] in the GOBP hierarchy. This GOBP term has 19 child terms, such as GO:0009409 [response to cold], GO:0009408 [response to heat], and GO:0009414 [response to water deprivation]. Since gene function becomes more and more specific as we move down the GOBP hierarchy, fewer and fewer genes have any given annotation. The result is that for specific types of stress, our data set contains many negatives and few positives. In the best case, for the term GO:0009613 [response to pest, pathogen or parasite], over 97% of the training data consists of negatives. The typical case is even worse. In fact, looking at all 19 types of stress, 5 types have no positives at all, and of the remaining 14 types, the median number of negatives is 99.2% of the training data. This highly unbalanced data made accurate prediction of gene function difficult. For this reason, we narrowed our study to the top stress term, GO:0006950 [response to stress]. To get positive training samples for this term, we propagated all genes in its offspring upward to it in the hierarchy. After up-propagation, the top stress term has 1,031 genes, or almost 9% of the total genes in the training data. The training data therefore contains 9% positives and 91% negatives.

Using GOBP terms to annotate all genes in A. thaliana is an ongoing project started in 2002 by TAIR 2728. The gene annotations (updated weekly) can be downloaded from TAIR 27. The predictions reported in this paper are based on the version for March 10, 2007. Using these annotations, we categorized the genes into annotated genes and unannotated genes. The annotated genes are those which have at least one GOBP annotation; the unannotated genes are those which have no GOBP annotations. In addition, a gene was treated as unannotated if its only annotation is the top GOBP category, GO:0008150 [biological process], since the function of such a gene is unknown. The result was 11,553 annotated genes and 11,193 unannotated genes in our data set.

The annotated genes formed the training data, in which a gene was called positive if it is annotated as a stress gene, and negative otherwise. The unannotated genes formed the prediction data. It should be noted that this approach probably introduces some false negatives into the training data, because genes not known to have a particular function are considered to be negative, even though future experiments could reveal them to have that function. That is to say, "unknown" is treated as "negative". However, the number of such false negatives should be small, since only a small number of genes participate in any given biological process. That is, most negatives are true negatives.

Using a variety of basic learning methods, we trained a number of classifiers to distinguish between genes that do and do not respond to stress, based on their patterns of gene expression in the training data. We then applied each classifier to the prediction data to estimate the function of the unannotated genes. In addition, we used cross validation to evaluate the performance of each classifier and to estimate the precision of each prediction.

We used five supervised learning methods: Logistic Regression (LR), Linear Discriminant Analysis (LDA), Quadratic Discriminant Analysis (QDA), Naive Bayes (NB) and K-Nearest Neighbors (KNN) 21 (see Methods). These methods were chosen because they are representative of the most basic supervised learning methods, the goal being to explore simple methods first. These methods are widely understood, take little computation time, and the results provide a benchmark against which more sophisticated methods can be compared. Moreover, as we show below, the results provided by these methods are good enough to enable biologists to conduct targeted laboratory experiments.

Each of the five methods is discriminative. That is, the classifiers learned by the methods assign a real number (called a discriminant value) to each gene, reflecting the classifier's certainty that the gene responds to stress. Formally, a discriminative classifier is a function, f^ MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGMbGzgaqcaaaa@2E11@, from genes to discriminant values. In our case, each gene is represented as a 290-dimensional vector, x, whose components are the expression levels of the gene under the 290 experimental conditions. Thus, if x is a vector representing a gene, then dv = f^ MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGMbGzgaqcaaaa@2E11@(x) is the discriminant value assigned to the gene by the classifier. Finally, a decision threshold, τ, is chosen, and the gene is predicted to respond to stress if and only if dv > τ.

To evaluate the performance of discriminative classifiers, it is common to use receiver operating characteristic (ROC) curves 32. A ROC curve plots the true positive rate (TP) of a classifier against the false positive rate (FP) for various decision thresholds. It therefore shows the quality of a classifier not at one threshold, but at many, and provides more information than a simple miss-classification rate (as in 33 for example). In practice, however, biologists are not usually interested in having more than a few dozen false positives, especially in unbalanced data such as ours, in which the number of false positives can rapidly overwhelm the number of true positives. We therefore use so-called ROC₅₀ curves 34, a variant of ROC curves in which the horizonal axis only goes up to 50 false positives. The area under a ROC₅₀ curve is the ROC₅₀ score 34, and is a measure of the overall usefulness of a classifier.

To estimate ROC₅₀ curves for our classifiers, we used 20-fold cross-validation (see Methods). Because cross-validation relies on a random split of the training data into folds (20 folds in our case), there is a certain randomness to the estimated ROC₅₀ curve. To provide more accurate results, we performed cross-validation ten times, each time with a different (randomly selected) 20-fold split of the data (see Methods). Each 20-fold split results in a slightly different ROC₅₀ curve. In some cases, we plot all ten of these curves, to give an idea of the uncertainty in classifier performance (Figure 1). In cases where this would result in overly cluttered graphs, we simply present the average of the ten ROC₅₀ curves (Figures 2 to 7, each of which show several average ROC₅₀ curves).

We generated ROC₅₀ curves for each supervised learning method combined with various amounts of dimensionality reduction. Using PCA, we reduced the original 290 dimensions to 5, 10, 15, 20, 40 and 100 dimensions, respectively. In this way, the original data set was transformed into six separate data sets of various dimensions. Each basic learning method (except KNN) was applied to the original data set and to each of the six reduced data sets. Thus, for each basic learning method (except KNN), we trained and tested seven different classifiers. In the case of KNN, we used only the original, unreduced data, but with five different values of K. Altogether, we trained and tested a total of 4 × 7 + 5 = 33 different classifiers. Figures 2 to 6 show the estimated performance of these basic classifiers. Each figure shows a number of ROC₅₀ curves, each derived using cross-validation averaged over a number of random splits of the data, as described above. Unlike traditional ROC curves, the axes of these curves give the number of true and false positives, instead of the proportion. The red dash-dot line near the bottom of each figure shows the expected performance of a random classifier (i.e., a classifier that ignores the expression data and guesses whether or not a gene responds to stress by essentially flipping a coin). The ROC₅₀ scores for the curves are shown in the legend of each figure.

As the figures show, in some cases the classifiers perform not much better than random, but in most cases they perform significantly better. The figures also show that the performance of each classification method depends heavily of the amount of dimensionality reduction used. Notice in particular that in some cases, the classifier trained on the reduced data has a much higher ROC₅₀ score than the classifier trained on the original, unreduced data. This is especially true for NB and QDA. For instance, the classifiers trained on the original data have low ROC₅₀ scores of 182.3 for NB and 115.2 for QDA. This is comparable to the random classifier, whose ROC₅₀ score is 122.5. However, reducing the dimensionality of the data to 15 increases their ROC₅₀ scores to 1373.1 and 1651.0, respectively. This shows the importance of dimensionality reduction. In contrast, KNN performs well for all the values of K that we used.

Figure 7 compares the basic classification methods by plotting the best performance of each. That is, for each of the basic classification methods, the ROC₅₀ curve with the highest ROC₅₀ score is reproduced in Figure 7. In addition, the figure shows the performance of a classification method that uses a weighted voting scheme (WV) to combine the 33 basic classifiers into a single, composite classifier. Notice that this composite classifier performs best of all. The next section describes how this composite classifier is constructed.

Combining different classifiers in prediction can be thought of as combining different opinions in decision making. The advantage is that a group opinion is better than a single opinion if the single opinions are correctly weighted and combined. In machine-learning systems, classifiers are often combined by weighted voting, in which the discriminant value of the combined classifier is a linear combination of the discriminant values of the individual classifiers. Formally, given a set of basic classifiers, f^1,…,f^M MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGMbGzgaqcamaaBaaaleaacqaIXaqmaeqaaOGaeiilaWIaeS47IWKaeiilaWIafmOzayMbaKaadaWgaaWcbaGaemyta0eabeaaaaa@3599@, and a set of weights, w₁, …, w_M, the combined classifier, f^ MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGMbGzgaqcaaaa@2E11@, is defined by the equation f^(x)=∑mwmf^m(x) MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGMbGzgaqcaiabcIcaOGqabiab=Hha4jabcMcaPiabg2da9maaqababaGaem4DaC3aaSbaaSqaaiabd2gaTbqabaGccuWGMbGzgaqcamaaBaaaleaacqWGTbqBaeqaaOGaeiikaGIae8hEaGNaeiykaKcaleaacqWGTbqBaeqaniabggHiLdaaaa@3EC3@. In our case, M = 33, as described above.

By judiciously choosing the weights, w₁, …, w_M, the performance of the combined classifier can be maximized. Various methods are available for doing this, such as model averaging and stacking 21. Using these methods on our data sets, we found that the ROC curve of the combined classifier was usually better than the ROC curves of the basic classifiers, as expected. Unfortunately, we also found that the ROC₅₀ curve of the combined classifier was usually worse (data not shown). We hypothesized that this is because our data sets are highly unbalanced. Intuitively, model averaging and stacking try to choose weights so as to correctly classify as much data as possible. In our case, this means trying to correctly classify the vast number of negative samples in our data sets, even if this means misclassifying the small number of positives. In other words, these methods try to minimize the total number of false positives, even though we only care about the first fifty.

To choose appropriate weights for our combined classifier, we used the heuristic that classifiers that perform well should be given more weight than classifiers that perform poorly. In our case, since we want to maximize the ROC₅₀ score of the combined classifier, we want to give high weight to classifiers with high ROC₅₀ scores. There are many ways to do this, but we found that it was sufficient to estimate and normalize the ROC₅₀ score of each basic classifier, and use this as its weight. That is, we used wm=s^m/∑ms^m MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqWG3bWDdaWgaaWcbaGaemyBa0gabeaakiabg2da9iqbdohaZzaajaWaaSbaaSqaaiabd2gaTbqabaGccqGGVaWldaaeqaqaaiqbdohaZzaajaWaaSbaaSqaaiabd2gaTbqabaaabaGaemyBa0gabeqdcqGHris5aaaa@3B09@, where s^m MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGZbWCgaqcamaaBaaaleaacqWGTbqBaeqaaaaa@2FBA@ is an estimate of the ROC₅₀ score of classifier f_m. Note that with these weights, if each f^m MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGMbGzgaqcamaaBaaaleaacqWGTbqBaeqaaaaa@2FA0@(x) is a number between 0 and 1 (as with our classifiers), then so is f^ MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGMbGzgaqcaaaa@2E11@(x). Also, this method automatically gives low weight to classifiers that use an inappropriate amount of dimensionality reduction, since such classifiers have low ROC₅₀ scores. In this way, the combined classifier incorporates not only the best combination of supervised learning methods, but also the best amounts of dimensionality reduction for each method.

To train and evaluate the combined classifier, we used two sets of validation data. After the basic classifiers were trained, one validation set was used to estimate their ROC₅₀ scores. The combined classifier was then constructed using these scores, and the second validation set was used to estimate its ROC₅₀ curve. Thus, the validation data for the basic classifiers is part of the training data for the combined classifier. To do this in a cross-validation setting, we used what amounts to nested cross-validation (see Methods). As shown in Figure 7, the resulting combined classifier has a higher ROC₅₀ score than any of the basic classifiers from which it is made.

Figure 1 gives another view of the performance of the combined classifier. Here, the thin dashed lines are a superposition of ten different curves, where each one is a different estimate of the combined classifier's true ROC₅₀ curve. As described earlier, each estimate of a classifier's ROC₅₀ curve includes some randomness, due to the random choice of folds during cross-validation. The ten dashed curves in Figure 1 are derived from ten different cross-validations, each one using a different set of folds. The thick solid line in the figure is the average of the other ten curves. Because averaging reduces variance, the average curve is a more accurate estimate of the true ROC₅₀ curve (i.e., has lower variance) than any of the other ten curves. The diagonal dash-dot line near the bottom of the plot shows the expected performance of a random classifier.

ROC and ROC₅₀ curves plot the number of true positives against the number of false positives. However, in applications such as ours, the precision is also of interest. Precision is the proportion of true positives (TP) among the predicted positives (PP). (It is also the complementary false discovery rate, 1-FDR 35.) Precision is important since each prediction is a potential experiment, and as a matter of economics, a biologist needs an estimate of how many of the experiments will succeed. This is especially important in situations, such as ours, where the number of real negatives is much greater than the number of real positives, and so there is a real possibility of having a huge number of failed experiments.

Figure 8 plots estimated precision against the number of predictions for the first hundred predictions. Notice that as the number of predictions increases (i.e., as the classifier's decision threshold is lowered), the precision decreases, meaning that fewer of the predictions are expected to be true. As in Figure 1, the thin dashed lines are a superposition of ten different curves, each one an estimate of the true precision curve, and the thick solid line is their average. Also, the horizontal dash-dot line near the bottom of the plot is the expected precision of a random classifier, and its height is equal to the ratio of the number of positives (i.e., stress genes) to the total number of samples (i.e., genes) in the training data. Since all the estimated precision curves are well above the horizontal dash-dot line, the performance of the combined classifier for the first hundred predictions is significantly better than random. Also, since Figures 1 and 8 show small variance, and since the variance of the average curves will be even less, the combined classifier should have stable prediction performance.

We trained the combined classifier on our Arabidopsis data set, using all 22,746 genes for Principal Components Analysis, and the 11,553 annotated genes for supervised learning, as described above. We then applied the classifier to the 11,193 unannotated genes, to get a set of 11,193 predictions (see Methods). Table 1 shows the top fifty predictions. Each row in the table is a prediction: the first (leftmost) entry is the rank of the prediction (1 being the top prediction); the second entry identifies a gene; the third entry is a discriminant value (measuring the likelihood that the gene responds to stress); and the fourth entry is the estimated precision of the prediction and all predictions above it (i.e., the fraction of these predictions expected to be true). As an example, consider the 23rd row of the table, the row for gene At1g09950. Since the estimated precision in this row is given as 0.7044, we expect that about 70% of the top 23 genes respond to stress, i.e., 16 genes.

Figures 9 and 10 provide visual evidence supporting these predictions. Each figure shows a heat map. These maps, known as "electronic Northerns" (or e-Northerns), were generated using the Expression Browser tool of the Botany Array Resource (BAR) and the AtGenExpress Stress Series (shoot) data set23. The program contains expression data for more than 22,000 genes across more than 1000 samples collected from NASCArrays, AtGenExpress Consortium, and the Department of Botany at the University of Toronto 22232436. Each row in an e-Northern is a gene, and each column is an experiment. The colour at a point represents the relative expression level of the gene during the experiment. More specifically, the colour represents the log₂ of the ratio of the average of replicate treatments relative to the average of corresponding controls. Yellow means that under the experimental conditions, the gene had the same expression level as the control. (The wide, yellow vertical stripes are the controls.) Red means that the gene had a higher expression level than the control (up-regulation), and blue means it had a lower expression level (down-regulation). A gene that shows significant up-regulation (or down-regulation) under stress conditions is likely to be involved in response to stress. Thus, unlike cross validation, electronic Northerns provide a means of evaluating the quality of predictions based on the prediction data, not just the training data. The e-Northerns of Figures 9 and 10, for instance, are based entirely on prediction data. In these e-Northerns, the experiments exposed the plant to various stress conditions, such as heat, cold, drought, UV-B radiation, etc. Figure 9 is the e-Northern for the top-50 predictions of our combined classifier, i.e., for the 50 genes predicted to most likely to respond to stress. For comparison, Figure 10 is the e-Northern for 50 genes chosen at random from the prediction set. Note that there is much more colour in Figure 9 than in Figure 10, especially red. This suggests that our combined classifier has indeed extracted meaningful gene expression patterns for genes that respond to stress.

From the predictions of the combined classifier, three genes were chosen for biological analysis using gene knockout experiments. Here, we present the results for one of these genes, At1g16850, which show it to be necessary for the normal response to temperature and NaCl. Our results also confirm that the other two genes, At1g11210 and At4g39675, are involved in a variety of stress responses (data not shown).

The criteria used to choose candidate genes for subsequent biological analysis were: 1) the gene must be expressed in either root or shoot, 2) gene expression should be strongly increased in response to abiotic stress, such as cold, drought, osmotic and salt stresses, 3) T-DNA knockout lines – in which a given gene's expression has been eliminated – should available from the Salk Institute 37, and 4) the gene should not have an annotated function nor be present in any patent database. Further bioinformatics analysis was performed using Athena for promoter motif prediction 38, Expression Angler for co-expressed gene analysis 22 and eFP browser for electronic representation of gene expression patterns 39.

The gene expression data from the Botany Array Resource at the University of Toronto contain detection calls: P (present), M (marginal) and A (absent). The detection call determines whether a transcript is reliably detected (present), partially detected (marginal), or not detected (absent). The following is an example for the gene At3g24440 under three selected conditions:

AT3G24440 : 243.10 P : 120.90 A : 109.40 M

We simply removed these detection calls (P, A, and M) in this study. In addition, gene expression levels were log transformed. The transformed data have approximately normal distributions while the raw data have approximately exponential distributions (data not shown). Many of the learning methods used in this study were designed with normal data in mind.

Each of the learning methods described below trains a discriminative classifier. We used the methods to train binary classifiers in which the two classes correspond to genes that respond to stress (Class 1) and genes that do not (Class 0). Given a vector, x, of gene expression measurements, each classifier returns a discriminant value, dv(x), reflecting the classifier's confidence that the gene belongs to Class 1. The gene is assigned to Class 1 if and only if dv(x) > τ, where τ is a decision threshold. For the classifiers LR, LDA, QDA and NB, the discriminate value is an estimate of p(k = 1|x), the posterior probability that the gene is in Class 1. For KNN, the discriminant value is simply a number between 0 and 1.

Hidden dependencies and noise among experiments may confound the classification problem. In particular, experiments that are biologically different may actually be similar in terms of gene expression. Principal components analysis (PCA) helps to identify independent information in the data by transforming it to a data set of reduced dimension. The attributes of the reduced data set, called principal components, explain most of the variance in the original data and are mutually uncorrelated and orthogonal 21. In addition, by reducing the dimension of the data, PCA reduces the number of parameters that must be estimated during supervised learning, thus permitting more efficient use of the data.

One can think of PCA as having a learning phase and a prediction phase. During learning, PCA is given a data set, from which it generates (learns) a linear transformation. This transformation maps high-dimensional vectors to low-dimensional vectors, and is applied to the given data set to reduce its dimensionality. During prediction, the transformation is applied to other data.

We used PCA to reduce the dimensionality of the gene expression data from its original 290 dimensions to p dimensions, for p = 5, 10, 15, 20, 40, 100. During learning, we gave PCA our entire data set of 22,746 genes, i.e., the 11,533 annotated genes and the 11,193 unannotated genes. This is possible because PCA is a form of unsupervised learning, so it uses only the gene expression measurements (which are known), and not the gene annotations (which are to be learned). This increases the effectiveness of PCA by doubling the amount of data that it uses during learning. That is, using a larger data set decreases the variance of the principal components learned by PCA, thus increasing their statistical significance and reducing the number of anomalous components.

It is worth noting that this use of PCA is different from that of many traditional applications of machine learning. This is because we apply PCA to the entire data set during learning, including the prediction data (i.e., the unannotated data). This is not possible in traditional applications simply because the prediction data is not known during learning. In such applications, a learning procedure is first trained and tested on one data set, and then applied to prediction data as it becomes available. This is not the situation for genome-wide expression experiments, since all the genes (and their expression levels) are known in advance, including the genes in the prediction set. PCA can therefore use both the prediction data and the training data during learning. This is a form of transductive inference 2931, in which the prediction data is known and exploited during learning.

After PCA is performed on the entire data set, supervised learning is performed on the annotated portion of the dimensionally-reduced data. (As described earlier, this is a form of semi-supervised learning 2930). The result is a set of classifiers, one for each supervised learning method. The classifiers are then applied to the unannotated portion of the dimensionally-reduced data to predict the missing annotations. Cross validation was used to estimate the accuracy of these predictions.

Before discussing our use of cross validation, we consider the simpler setting in which the annotated data is divided into two parts, training data and validation data 21. This will clarify our handling of PCA during validation. In this setting, classifier evaluation proceeds as follows. First, PCA is applied to the entire data set (training, validation and prediction data) to produce a dimensionally-reduced data set. Then, a supervised learner uses the (dimensionally-reduced) training data to produce a classifier. Finally, the accuracy of the classifier is estimated using the (dimensionally-reduced) validation data. Note that this process treats the validation and prediction data equally. That is, they are both used during unsupervised learning, and neither is used during supervised learning. In this way, the validation data is representative of the prediction data, as it should be. Also note that PCA is now effectively a preprocessing phase prior to supervised learning.

Because PCA is applied to the entire data set, this validation process estimates the accuracy of the classifier on prediction data that is known and used during learning. In particular, the estimate does not apply to new prediction data that might arrive in the future (e.g., if new genes were discovered). In fact, it would likely be an overestimate of classifier accuracy on such data. However, this is not an issue in our application, since most if not all of the genes in Arabidopsis are already known. Moreover, even if some new genes were to be discovered in the future, we could simply add them to our prediction data and retrain the classifier on the enlarged data set.

The above ideas are easily extended to cross validation. First, PCA is applied to the entire data set. Then, a supervised learner uses the annotated portion of the dimensionally-reduced data to produce a classifier. Finally, this classifier is evaluated by cross validation in the normal way, as described below. Note that this approach has the added computational advantage that PCA is applied only once, to the entire data set, and not over-and-over again during the many training phases of cross validation. The discussions below assume that the entire data set has been preprocessed using PCA, so that all references to data refer to the dimensionally-reduced data. Also, all references to generalization performance refer to the accuracy of the classifier on the given set of prediction data.

We used 20-fold cross-validation to assess the generalization performance of each classifier as well as to estimate the precision of its predictions. We randomly divided the annotated data into 20 non-overlapping, equal-sized parts, called folds. The classifier was trained on 19 of these folds, and tested on the remaining fold; i.e., the trained classifier was used to generate a discriminant value for each gene in the remaining fold. This was done in all 20 possible ways, using a different testing fold each time. In this way, a discriminant value, dv, was generated for every gene in the training set. Each gene in the training set was then predicted to be positive (i.e., to respond to stress) if and only if dv > τ, where τ is a decision threshold. From these predictions, true and false positives were computed, from which a point on the ROC₅₀ curve was plotted. Using a large number of different decision thresholds, we plotted a large number of points on the ROC₅₀ curve, effectively generating the entire curve. The area under this curve is the ROC₅₀ score. To get an idea of how stable the estimated performance of the classifier is, we repeated the entire cross-validation and curve-generation procedure 10 times, each time using a different, random, 20-fold split of the training data.

The above procedure was applied to all the basic classifiers, but assessing the combined classifier involved an additional subtlety. Recall that the combined classifier is a linear combination of the basic classifiers, where the weight given to a basic classifier is proportional to its estimated ROC₅₀ score. The subtlety is in computing that score. A naive approach would be to simply use the above procedure to compute a ROC₅₀ score for each basic classifier. However, this would mean that during cross validation, 19 of the 20 folds are used to train the basic classifiers, while the 20^th fold is used to compute the ROC₅₀ scores. The result is that all 20 folds are involved in computing the weights. Thus, all 20 folds are involved in constructing (i.e., training) the combined classifier, so no folds are left for testing it. If cross validation were used anyway to assess the combined classifier, it would amount to using training data as testing data, and the results would tend to overestimate the classifier's performance.

As described earlier, we surmount this problem by using two sets of validation data. Loosely speaking, 18 of the 20 folds are used to train the basic classifiers, a 19^th fold is used to compute their ROC₅₀ scores, and the 20^th fold is used to test the combined classifier. This results in what might be called nested cross validation. To start, the training data are divided randomly into 20 folds. Picking one of these as a testing fold, the other 19 are used to train the combined classifier. This in turn involves 19-fold cross validation to train and test the basic classifiers (and compute their ROC₅₀ scores). Thus, each time the combined classifier is trained once, the basic classifiers are trained 19 times. Since the combined classifier is trained 20 times, each basic classifier is trained a total of 20 × 19 = 380 times. A similar form of nested cross validation is involved in Stacking 21.

To predict which genes respond to stress, we first train a combined classifier using the 11,553 annotated genes in the training data. The classifier is then applied to the 11,193 unannotated genes in the prediction data. After this step, each annotated gene has a discriminant value, dv. The unannotated genes are then sorted in descending order by discriminant value, as illustrated in Table 1. To make actual predictions, a gene in the sorted list is chosen as a decision point. This gene and every gene above it in the sorted list are then predicted to respond to stress. In other words, suppose dv is the discriminant value of the chosen gene. An unannotated gene is then predicted to respond to stress if and only if its discriminant value is at least dv. The fraction of these predictions that are true is the precision of the predictions. We estimate this precision using the training data. Recall that each gene in the training set has a discriminant value assigned to it during cross validation. We also know which of these genes respond to stress. To estimate the precision of our predictions, we look at those genes in the training set whose discriminant value is at least dv. The fraction of them that respond to stress is an estimate of precision.

Using this idea we actually get ten precision estimates, not one. This is because we do cross validation ten times, using ten different random splits of the data. The result is that each gene in the training set receives ten discriminant values, and for each one we get a different precision estimate. We could simply use the average of these ten precision estimates; however, to reduce the variance of the estimate, we use a weighted average. Specifically, let us number the cross validation runs from i = 1, …, 10. Then, given a discriminant value, dv, let PP_i be the number of genes in the training set whose discriminant values is at least dv in the i^th run of cross validation. (These are the predicted positives.) Let TP_i be the number of these genes that respond to stress (the true positives). Using only this cross validation run, the estimated precision would be TP_i/PP_i. One problem with this estimate is that if dv is high, then PP_i (and hence TP_i) could be 0, so the precision estimate would be undefined, something we observed frequently in practice. More generally, if PP_i (and hence TP_i) is low, then the precision estimate will have high variance, since it is supported by very little data. To circumvent these problems, we estimate the precision using the formula

p r e c i s i o n = ∑ i TP i ∑ i PP i = ∑ i w i × TP i PP i MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqWGWbaCcqWGYbGCcqWGLbqzcqWGJbWycqWGPbqAcqWGZbWCcqWGPbqAcqWGVbWBcqWGUbGBcqGH9aqpdaWcaaqaamaaqababaGaeeivaqLaeeiuaa1aaSbaaSqaaiabdMgaPbqabaaabaGaemyAaKgabeqdcqGHris5aaGcbaWaaabeaeaacqqGqbaucqqGqbaudaWgaaWcbaGaemyAaKgabeaaaeaacqWGPbqAaeqaniabggHiLdaaaOGaeyypa0ZaaabuaeaacqWG3bWDdaWgaaWcbaGaemyAaKgabeaakiabgEna0oaalaaabaGaeeivaqLaeeiuaa1aaSbaaSqaaiabdMgaPbqabaaakeaacqqGqbaucqqGqbaudaWgaaWcbaGaemyAaKgabeaaaaaabaGaemyAaKgabeqdcqGHris5aaaa@59BB@

where wi=PPi/∑jNPPj MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqWG3bWDdaWgaaWcbaGaemyAaKgabeaakiabg2da9iabbcfaqjabbcfaqnaaBaaaleaacqWGPbqAaeqaaOGaei4la8YaaabmaeaacqqGqbaucqqGqbaudaWgaaWcbaGaemOAaOgabeaaaeaacqWGQbGAaeaacqWGobGta0GaeyyeIuoaaaa@3DCF@. The right-hand formula is a weighted average of individual precision estimates, TP_i/PP_i. It gives more weight to precision estimates that are based on more data, i.e., for which PP_i is higher. In addition, by using the left-hand formula, we rarely end up dividing by zero, since the denominator is a sum of (random) non-negative numbers; i.e., Σ_iPP_i is much less likely to be zero than is any individual PP_i.

Wild type and homozygous mutant seeds were plated on 0.5X MS media. They were stratified for 3 days and then germinated at 25C for 7 days. The abiotic temperature stresses consisted of 7 days exposure to either 30C, 14C or 4C. Anthocyanin levels were quantified as a measure of plant stress response. Anthocyanin was extracted using methanol-HCl 43. In order to measure response to salt stress, plants were germinated for 3 days on 0.5X MS media and then transferred to medium containing 50 mM NaCl or to control plates. New root growth was measured 7 days after the transfer.