EGASP: the human ENCODE Genome Annotation Assessment Project

gb-2006-7-s1-s2 GBJ Review EGASP: the human ENCODE Genome Annotation Assessment Project Guigó Roderic rguigo@imim.es Flicek Paul Abril F Josep Reymond Alexandre Lagarde Julien Denoeud France Antonarakis Stylianos Ashburner Michael Bajic B Vladimir Birney Ewan Castelo Robert Eyras Eduardo Ucla Catherine Gingeras R Thomas Harrow Jennifer Hubbard Tim Lewis E Suzanna Reese G Martin mreese@omicia.com

Centre de Regulació Genòmica, Institut Municipal d'Investigació Mèdica-Universitat Pompeu Fabra, E08003 Barcelona, Catalonia, Spain

European Bioinformatics Institute, Wellcome Trust Genome Campus, Hinxton, Cambridge CB10 1SD, UK

Center for Integrative Genomics, University of Lausanne, Switzerland

University of Geneva Medical School and University Hospitals of Geneva, 1211 Geneva, Switzerland

Department of Genetics, University of Cambridge, Cambridge CB3 2EH, UK

South African National Bioinformatics Institute (SANBI), University of Western Cape, Bellville 7535, South Africa

Affymetrix Inc., Santa Clara, California 95051, USA

Wellcome Trust Sanger Institute, Wellcome Trust Genome Campus, Hinxton, Cambridge CB10 1SA, UK

Department of Molecular and Cellular Biology, University of California, Berkeley, California 94792, USA

Omicia Inc., Christie Ave., Emeryville, California 94608, USA

Member of the EGASP Organizing Committee

Member of the EGASP Advisory Board

Genome Biology EGASP '05: ENCODE Genome Annotation Assessment Project Roderic Guigó, Martin G Reese Research 1465-6906 2006 7 Suppl 1 S2 http://genomebiology.com/2006/7/S1/S2 16925836 10.1186/gb-2006-7-s1-s2 7 8 2006 2006 BioMed Central Ltd.

Abstract

Background

We present the results of EGASP, a community experiment to assess the state-of-the-art in genome annotation within the ENCODE regions, which span 1% of the human genome sequence. The experiment had two major goals: the assessment of the accuracy of computational methods to predict protein coding genes; and the overall assessment of the completeness of the current human genome annotations as represented in the ENCODE regions. For the computational prediction assessment, eighteen groups contributed gene predictions. We evaluated these submissions against each other based on a 'reference set' of annotations generated as part of the GENCODE project. These annotations were not available to the prediction groups prior to the submission deadline, so that their predictions were blind and an external advisory committee could perform a fair assessment.

Results

The best methods had at least one gene transcript correctly predicted for close to 70% of the annotated genes. Nevertheless, the multiple transcript accuracy, taking into account alternative splicing, reached only approximately 40% to 50% accuracy. At the coding nucleotide level, the best programs reached an accuracy of 90% in both sensitivity and specificity. Programs relying on mRNA and protein sequences were the most accurate in reproducing the manually curated annotations. Experimental validation shows that only a very small percentage (3.2%) of the selected 221 computationally predicted exons outside of the existing annotation could be verified.

Conclusion

This is the first such experiment in human DNA, and we have followed the standards established in a similar experiment, GASP1, in Drosophila melanogaster. We believe the results presented here contribute to the value of ongoing large-scale annotation projects and should guide further experimental methods when being scaled up to the entire human genome sequence.

Background

During the first decade of the 21st century the sequencing of whole genomes has become a routine biological practice. The list of chordates with assembled genome sequences now numbers nearly two dozen, while the total number of sequenced bacteria, archea, and eukaryota is approaching 2,000. The genome sequence is said to be an organism's blueprint: the set of instructions dictating its biological traits. In higher eukaryotic organisms, however, these traits are apparently encoded by only a small fraction of the genome sequence that is functional (possibly less than 5% in the case of the human genome). The genes are a major component of this functional sequence. While there is growing evidence for many functional non-protein coding RNA genes, such as miRNAs and snoRNAs, the largest and best studied subset of the human genes comprise the protein coding genes, genes specifying the amino acid sequence of the proteins. Thus, locating the genes in a newly sequenced genome is a first, essential step toward understanding how the organism translates its genome sequence into biological function. This paper focuses on the identification of protein coding genes, if not otherwise noted.

Maybe to the surprise of many, five years after the first drafts of the human genome sequence became available 12, and nearly three years after the announcement of the completion of the sequencing 3, a complete set of protein coding genes encoded in the human genome does not exist. One reason for the lack of a complete gene set is that an appropriately rigorous standard has been set for the human genome: every gene, exactly correct. And as shown in this paper, only very few of the human genes seem to be missing from the computational predictions, but the exact genomic structure of these genes is estimated to be correct for only 50% of the predicted genes. In other words, only very few protein coding genes appear to have been totally missed today. Nevertheless, getting the entire genomic structure of a protein coding gene right is still a very difficult task, compounded by the large amount of alternative splicing characterizing human genes. Our assessment here tries to quantify the status of these differences in the current human genome annotations and computational prediction programs.

Automatic genome annotation methods

To date, accurate automatic annotation of the human genome (and of other genomes with significant cDNA libraries) strongly relies on an elaborate mapping of these known gene sequences onto the genome sequence. This method of genome annotation requires high quality and a nearly complete set of cDNA sequences. Datasets trying to achieve this goal, but are still works in progress, are the RefSeq database 4 and those currently being produced by the Mammalian Gene Collection (MGC) 5. As the MGC project - and similar efforts to deepen the coverage of the fraction of the human genome being transcribed - continues, cDNA mapping based gene identification methods are becoming increasingly accurate. While few organisms will have the rich cDNA libraries that are currently being developed for the human genome, the availability of protein sequence data from evolutionarily close relatives has been effectively used in addition to cDNA data for automatic gene prediction across many of the currently sequenced mammals. The most commonly used annotation pipelines are the ENSEMBL pipeline 6, the UCSC genome browser's 7 Known Genes (KG II) pipeline, and the Gnomon pipeline at the NCBI 8. It remains unclear, however, what fraction of the low and specifically expressed transcripts and of alternatively spliced isoforms can be effectively recovered from cDNA libraries. Additionally, orthologous proteins from other species may not align genes that are rapidly evolving. For these reasons, current cDNA and protein-based methods are likely to provide an incomplete picture of the protein coding gene content of the human genome. These methods will be less accurate for genomes with fewer expressed sequences and comparative options.

For automatic annotation of genomes without deep expressed sequence libraries, any available cDNA or expressed sequence tag (EST) based annotation is often complemented by dual (or multiple) genome comparative predictions. These predictions are obtained by means of the analysis of the patterns of sequence conservation between genome sequences of evolutionarily related organisms. As examples, programs such SGP2 9, SLAM 1011 and TWINSCAN 1213 have contributed efficiently to the annotation of a number of vertebrate genomes, including mouse 14, rat 15, and chicken 16. This type of comparative-based automatic gene prediction can produce highly accurate gene sets when the sequence of related species is available, but few ESTs have been sequenced, such as the case with the fungus Cryptococcus neoformans 17.

Occasionally, the so-called single genome ab initio predictors - programs that use statistical sequence patterns, such as the coding reading frame, codon usage or splice site consensus sequences, for gene identification - are also used to complement cDNA and comparative based methods. When no genome exists at the appropriate phylogenetic distance, and the cDNA or EST coverage of the transcriptome is shallow, single genome ab initio predictions play an important role in genome annotation, such as those obtained, for example, by the programs GENSCAN 18 and GENEID 19 in the initial annotation of the genome of the fish Tetraodon nigroviridis 20.

In summary, despite substantial progress in the past decade and the existence of highly accurate gene sets in a number of organisms, current gene identification methods are, as yet, not able to produce a complete catalogue of the set of protein coding genes in higher eukaryotic genomes (see 21 for a recent review).

Assessing the accuracy of automatic genome annotation

Over the past quarter century, a large number of automated gene prediction algorithms have been introduced, which can be loosely grouped based on the general strategies described above. These methods vary widely in the details of their implementation and in the number and location of predicted protein coding genes. Thus, the issue of evaluating the accuracy of the predictive methods has been recurrent within the field of computational gene prediction. The early work of Burset and Guigó 22, and the subsequent analysis of Bajic 23, Baldi et al. 24, Guigó et al. 25, Rogic et al. 26 and others, provide a framework - a set of metrics and a protocol - to consistently evaluate gene prediction methods. Essentially, a set of well-annotated sequences are used as a test set. The gene prediction programs are run on these sequences, and the predictions obtained are compared with the annotations. A number of measures are computed to evaluate how well the predictions reproduce the annotation. Typically, predictions are evaluated at nucleotide, exon and gene levels. At all three levels, two basic measures are computed: sensitivity, the proportion of annotated features (nucleotide, exon, gene) that have been predicted; and specificity, the proportion of predicted features that is annotated. One problem with this approach is that, until recently, very few large genomic sequences were well annotated and only the coordinates of the coding exons within a gene could be considered. Moreover, because methods did not exist to predict alternative splicing, the test sets used to evaluate computational gene predictions consisted of a few hundred short sequences encoding single genes from which alternatively spliced isoforms had been removed. This led to an oversimplification of the problem and, in turn, to an overestimation of the real accuracy of the programs 25. Furthermore, many programs were developed in-house and were, therefore, not accessible for independent evaluation.

To address the problem of independent, objective assessment of the state-of-the-art in automated tools and techniques for annotating large contiguous genomic DNA regions and eventually complete genomes, a first Genome Annotation Assessment Project (GASP1) was organized in 1999 27. In many ways, GASP1 was set up similarly to CASP (Critical Assessment of Techniques for Protein Structure Prediction) 28. In short, at GASP1, a genomic region in Drosophila melanogaster, including auxiliary training data, was provided to the community and gene finding experts were invited to send the annotation files they had generated to the organizers before a fixed deadline. Then, a set of standards were developed to evaluate submissions against the later published annotations 29, which had been withheld until after the submission stage. Next, the evaluation results were assessed by an independent advisory team and publicly presented at a workshop at the Intelligent Systems in Molecular Biology (ISMB) 1999 meeting. This community experiment was then published as a collection of methods and evaluation papers in Genome Research 27.

The ENCODE Genome Annotation Assessment Project

Inspired by GASP1, and within the context of the ENCyclopedia Of DNA Elements (ENCODE) project, we organized the ENCODE GASP (EGASP) community experiment, which followed closely the model of its predecessor, GASP1 27. The ENCODE project was launched two years ago by the National Human Genome Research Institute (NHGRI) with the aim of identifying all functional elements in the genome sequence through the collaborative effort of computational and laboratory-based scientists 30. The pilot phase of the project is focused on a selected 30 Mb of sequence within 44 selected regions (Table 1) across the human genome, which represents approximately 1% of the genome sequence.

Table 1

The 44 selected sequences within the ENCODE region

Random picks Mouse homology

Sequence Set

Manual picks

Low

Medium

High

Gene density

Training

ENm006

ENr132

ENr231

ENr333

High

ENr232

ENr334

ENm004

ENr222

ENr323

Medium

ENr223

ENr324

ENr111

Low

ENr114

Test

ENm002

ENr131

ENr233

ENr331

High

ENm005

ENr133

ENr332

ENm007

ENm008

ENm009

ENm010

ENm011

ENm001

ENr121

ENr221

ENr321

Medium

ENm003

ENr122

ENr322

ENm012

ENr123

ENm013

ENm014

ENr112

ENr211

ENr311

Low

ENr113

ENr212

ENr312

ENr213

ENr313

ENCODE sequences were assigned to either the training or the test set based on annotation data availability (see the section 'The EGASP experiment'). For the performance evaluation, only the test set sequences were used. The numeric code for the randomly picked sequence names correspond to the non-exonic conservation with the mouse genome, the density of previously identified genes, and the sequence number, respectively; numbers vary from 1 (low), to 3 (high). Manually selected sequences range in size from 500 kbp to 2 Mbp, while random regions are 500 kbp. The selection and stratification criteria for all the sequences is described at the ENCODE project web site [34].

Within the ENCODE project, the GENCODE consortium 31 was set up. This group, in collaboration with the HAVANA team 32 at the Sanger Institute, has produced a high quality annotation of the gene content of the ENCODE regions through a combined manual, computational and experimental strategy 33. The EGASP experiment was organized with the main goal of evaluating how well automatic methods are able to reproduce this annotation produced by GENCODE. A second goal of EGASP was to assess the completeness of the GENCODE annotation and, in this regard, EGASP was designed such that, in a follow-up step, a number of computational gene predictions not included in GENCODE were tested experimentally.

In what follows, we first describe the organization and structure of the EGASP experiment. We then present the results of the evaluation of the submitted predictions against the GENCODE annotation, and finally we present the results of the experimental verification of the novel predictions.

The EGASP experiment

Data: the benchmark sequence of 44 selected ENCODE regions

Description of the sequence

The 44 ENCODE regions represent 30 Mb (approximately 1%) of the human genome 30. Approximately half of the sequence corresponds to a set of 14 manually selected regions including well-studied genes and for which a significant amount of prior comparative sequence data was available. The remaining 30 genomic regions were chosen based on a stratified random sampling based on two measures: gene density (from previous annotations) and non-exonic conservation with the mouse genome sequence. Briefly, each portion of the human genome sequence was classified as high, medium, or low if it fell in the top 20%, the middle 30%, or the bottom 50%, respectively, of the above two measures. Several 500 kb sequences were chosen from each of the nine classifications created by this stratification procedure.

Table 1 lists the 44 selected sequences within the ENCODE region and classifies them based on random/manual selection, previously known gene density and non-exonic conservation to the mouse genome. It also describes the size differences between the sequences. Information about the criteria used to select the regions and their characteristics can be found on the ENCODE website 34. The sequences of the ENCODE regions (as well as multiple functional annotations) can be downloaded from the UCSC ENCODE browser 35.

We defined the sequences used for the EGASP experimental evaluation by taking advantage of the prior work of the HAVANA team 32, which had previously comprehensively annotated and released annotation for several human chromosomes 36373839404142. Updated annotation for the 13 ENCODE regions on these chromosomes was released in January 2005 as a 'training' set for the EGASP experiment. The manual annotation of the other 31 ENCODE regions was held back from release until after the automated gene predictions had been received. The 31 EGASP test regions represent a total of 21.6 million base-pairs (bp) of sequence. Further information is available at the GENCODE website 31.

The reference gene set: the GENCODE annotations

The ENCODE regions had been subjected to an exhaustive annotation strategy prior to EGASP by the HAVANA team. In short, the annotators initially build coding transcripts manually based on alignments of known mRNA, EST and protein sequences to the human genome. The initial gene map delineated in this way was then experimentally refined through reverse transcription (RT)-PCR and rapid amplification of cDNA ends (RACE), which essentially confirmed the existence of the mRNA sequences of the hypothesized genes. Finally, the initial annotation was refined by the annotators based on these experimental results. While the initial annotation by the HAVANA team is augmented by some experimentally verified ab initio and dual-genome gene predictions without a priori transcript sequence support, these constitute a marginal fraction of the entire GENCODE annotation set. The strategy is described in detail elsewhere in this issue 33. We used this final annotation as the reference set for EGASP, and refer to it as the GENCODE annotation.

The protein coding GENCODE annotation for all 44 ENCODE regions consists of 2,471 total transcripts representing 434 unique protein coding gene loci. There are 1,097 coding transcripts that code for 993 unique proteins. The annotation identifies 5.7 total transcripts per locus, with an average of 2.52 coding transcripts. Of the 434 coding loci, 393 contain multi-exon transcripts. In line with earlier estimates 43, 86% of the multi-exon loci exhibit alternative splicing in either the coding or non-coding transcripts. Sixty percent of multi-exon loci have alternative coding transcripts. See 33 in this issue for additional details.

Incomplete annotation

The GENCODE annotation includes incomplete genes and transcripts. These are caused both by the truncation of some features at the end of the ENCODE regions and by transcript annotations that may be incomplete due to lack of evidence. In the rare case that an exon crossed an ENCODE region boundary, the exon was truncated at the ENCODE region boundary in both the annotations and the predictions to ensure that the nucleotide level evaluation statistics were computed correctly (see Materials and methods).

EGASP: a community experiment

To determine an automatic method's ability to reproduce the GENCODE annotation, we organized EGASP in the following way: In January 2005, the GENCODE annotation for 13 of the 44 ENCODE regions (the 'training regions' defined above) was publicly released. With the release of this annotation, EGASP was officially announced: gene and other DNA feature prediction groups world-wide were asked to submit genome annotations on the remaining 31 ENCODE regions, for which the GENCODE annotations would not be released until the deadline for submission expired. Participating groups had access to the annotation of the 13 training regions, as well as to the sequences and all additional publicly available data for all 44 ENCODE regions. No other pre-defined and pre-selected auxiliary data, such as cDNA databases, EST sequences or other genome alignments, were given to the submitters. However, many of the 31 test regions had been previously and extensively annotated by other groups. For example, ENm001, the greater cystic fibrosis transmembrane receptor (CFTR) region, has been extensively studied 44.

Participants were asked to submit their genome annotations on the 31 ENCODE test regions, using whatever methods and data were available to them. To be able to better compare different DNA feature prediction methods, we predefined the following prediction categories and asked the submitters to indicate in which category they were submitting: methods using any type of available information; single-genome ab initio methods; EST-, mRNA-, and protein-based methods; dual- or multiple-genome based methods; methods predicting unusual genes (non-canonical splicing, short intron-less genes, and so on); and exon-only predictions.

Finally, we allowed an extra category (category 7) for methods predicting other annotation features, including pseudogenes and promoters. Bajic et al. 45 have conducted a comprehensive evaluation of the promoter predictions and see Zheng and Gerstein 46 for a paper on pseudogenes.

A web server (Figure 1) 47 was set up to collect all the submissions and each group was able to submit predictions for more than one category. The submitted predictions, as well as the GENCODE annotations for the test sequence set, were kept confidential until the submission deadline on 15 April 2005. The format for submissions was the Gene Transfer Format (GTF) 48. An advisory committee (Table 2) was formed to oversee the submission and evaluation processes and provide advice for the evaluation.

Figure 1

A screenshot of the EGASP submission server [47]

A screenshot of the EGASP submission server [47]. The server was user-authenticated in order to keep the submitted predictions in private before the EGASP workshop. Initially, there were eight suggested submission categories. However, after the workshop, category 5 was not used at all and removed. Promoter and pseudogene predictions from category 8 were then kept as a new category 7, which is not analyzed in this paper (see [45] instead).

Table 2

EGASP organizing and advisory committees

Organizers

Advisory board

Jennifer Ashurst (Wellcome Trust Sanger Institute)

Michael Ashburner (Cambridge University)

Ewan Birney (European Bionformatics Institute)

Vladimir B Bajic (Institute for Infocomm Research)

Peter Good (National Human Genome Research Institute)

Tom Gingeras (Affymetrix, Inc.)

Roderic Guigó (Institut Municipal d'Investigació Mèdica)

Suzanna Lewis (Berkeley)

Tim Hubbard (Wellcome Trust Sanger Institute)

Martin Reese (Omicia, Inc.)

By the submission deadline on 15 April 2005, 18 groups had submitted 30 prediction sets (Table 3). All the submitted predictions together with the annotations are available through the GencodeDB Genome Browser (Figure 2) 49, as well as through the UCSC Genome Browser ('EGASP' tracks). They can also be downloaded from the ftp server as plain text GTF files 50.

Table 3

Summary of programs used to determine predictions submitted for each EGASP category

Submission category

Program

Affiliation

Reference

1 (AUGUSTUS-any)

AUGUSTUS

Georg-August-Universität, Göttingen

[58]

2 (AUGUSTUS-abinit)

3 (AUGUSTUS-EST)

4 (AUGUSTUS-dual)

FGENESH++

Softberry Inc.

[56]

JIGSAW

The Institute for Genomic Research (TIGR)

[59]

1 (PAIRAGON-any)

PAIRAGON and NSCAN_EST

Washington University, Saint Louis (WUSTL)

[57]

3 (PAIRAGON+NSCAN_EST)

GENEMARK.hmm

Georgia Institute of Technology

[60]

GENEZILLA

TIGR

[81]

ACEVIEW

National Center for Biotechnology Information (NCBI)

[52]

ENSEMBL

The Wellcome Trust Sanger Institute (WTSI) and

[64]

European Bioinformatics Institute (EBI)

EXOGEAN

Ecole Normale Superieure, Paris

[62]

EXONHUNTER

University of Waterloo

[63]

ACESCAN*

Salk Institute

[82]

DOGFISH-C

WTSI

[67]

NSCAN

WUSTL

[57]

SAGA

University of California at Berkeley

[66]

MARS

WUSTL - EBI

[65]

GENEID-U12

Institut Municipal d'Investigació

SGP2-U12

Mèdica, Barcelona

ASPIC^†

Università degli Studi di Milano

[83]

6 (AUGUSTUS-exon)

AUGUSTUS

Georg-August-Universität, Göttingen

[58]

CSTMINER^‡

Università degli Studi di Milano

[84]

DOGFISH-C-E^§

WTSI

[67]

SPIDA

EBI

[85]

UNCOVER^§

Duke University

[86]

CCDSGene

UCSC tracks [7]

[55]

KNOWNGene

[54]

REFSEQ (REFGene)

[4]

GENEID

[19]

GENSCAN

[18]

ACEMBLY

[52]

ECGene

[53]

ENSEMBL (ENSGene)

[6]

MGCGene

[5]

SGP2

[9]

TWINSCAN

[12,13]

CODING 20050607

GENCODE annotation

[33]

GENES 20050607

A complete listing of the number of features for each sequence obtained by each method is available at the Supplementary material web page [51]. *The ACESCAN group submitted results only for the training set and, therefore, has not been evaluated. ^†ASPIC only provided results for the training regions and, therefore, has not been evaluated. Moreover, ASPIC submitted only intron annotations and should be considered in category 6. ^‡CSTMINER predicts coding regions but does not provide strand information. ^§DOGFISH-C-E and UNCOVER predict only novel exons; this makes it difficult to compare these methods with the others in the same category.

Figure 2

The GencodeDB Genome Browser

The GencodeDB Genome Browser. A screenshot of the GencodeDB Genome Browser [49], displaying the annotation features on 100 Kbp from the ENm001 region (chr7: 116,074,892-116,174,891). The annotations along with the predicted genes by each submitted method were made publicly available together with further experimental evidence, such as TARs/transfrags.

Predictions were compared with the reference set GENCODE annotations and assessed by members of the advisory and organizing committees (Table 2), all selected as independent experts in this field. The results of this assessment were presented at a workshop that took place at the Wellcome Trust Genome Campus in Hinxton, UK, on 6 and 7 May 2005. The advisory and organizing committees met on 4 May for a pre-evaluation of the predictions, and to determine a number of summary statistics. Each of the submitting groups was invited to present their methods and submissions at the workshop with a focus on what went right and what went wrong. In total, 16 groups were represented at the workshop. The final prediction evaluation results from the workshop are discussed in the next section.

Results

The evaluation of the predictions against the annotation

The protocol to evaluate the predictions

The main goal of the EGASP experiment was to evaluate the ability of automatic methods of genome annotation to reproduce the manual and experimental annotation of the ENCODE regions described above. By this standard, a perfect prediction strategy would produce annotation completely consistent with the GENCODE annotation.

For the purposes of evaluating the submitted predictions, we considered only the results for the 31 test ENCODE regions, which were the 'blinded' regions for which no GENCODE annotations were available during the submission phase. Potential biases introduced by this restriction will be addressed below. The statistics reported are computed globally for the test region, which means that the total number of prediction successes and failures for all 31 regions are compared directly to the total number of annotated exons, transcripts and genes for all 31 regions.

We evaluated each set of submitted predictions at four distinct levels: nucleotide accuracy, exon accuracy, transcript accuracy, and gene accuracy. At the earlier GASP1 workshop, transcript accuracy levels were not assessed due to the limited transcript information and the lower levels of alternatively spliced transcripts in Drosophila melanogaster 27. For this study we also made a distinction between the statistics calculated for the coding portions of the mRNA transcripts (coding sequence (CDS) evaluations) and the mRNA transcripts as a whole (mRNA evaluations).

For each of the four levels, we calculated the sensitivity and specificity of the predictions as defined below. In some cases, we have also computed other standard measures previously used in the gene finding literature (see 222324252627). Many additional measures of accuracy have been computed on the EGASP predictions, and they are available through the Supplementary Material web page 51.

Non-EGASP entries

To compare the EGASP results to existing community standards, we also evaluated the performance of 11 gene annotation tracks published in the UCSC Browser 7 just before the start of the EGASP workshop. These tracks included two single genome ab initio prediction methods (GENSCAN 18 and GENEID 19) and two dual-genome prediction methods (TWINSCAN 1213 and SGP2 9). We also considered four methods we classified as using expressed sequence (ENSGENE 6, ACEMBLY 52, MGCGENES 5, and ECGENE 53) and three we classified as using any information (UCSC 'KNOWN' genes 54, REFSEQ genes 4, and CCDSGENES 55).

Measures used for evaluating predictions: definitions

Nucleotide level accuracy is a comparison of the annotated nucleotides with the predicted nucleotides. Individual nucleotides appearing in more than one transcript in either the annotation or the predictions are considered only once for the nucleotide level statistics (Figure 3a). Nucleotide predictions must be on the same strand as the annotations to be counted as correct. At the nucleotide level, sensitivity (Sn) is the proportion of annotated nucleotides (as being coding or part of an mRNA molecule) that is correctly predicted, and specificity (Sp) the proportion of predicted nucleotides (as being coding or part of an mRNA molecule) that is so annotated. As a summary measure, we have computed either the simple average of these two measures, or the correlation coefficient between the annotated and the predicted nucleotides (see 222324252627).

Figure 3

Gene Feature Projection for evaluation

Gene Feature Projection for evaluation. The process of projecting genic features into unique nucleotide and exon coordinates in order to compute the accuracy values (see text for details).

The exon level accuracy is calculated with the requirement that an exon in the prediction must have identical start and end coordinates as an exon in the annotation to be counted correct. Only the unique exons in each set are considered (see Figure 3b for a graphical example of how unique exons are collected from both the annotation and prediction sets; also see 222324252627 for more details on these definitions). At the exon level, sensitivity is computed as the proportion of annotated exons correctly predicted, and specificity as the proportion of predicted exons that is annotated. As a summary measure, we have computed the average of these two measures. In addition, we have computed 'missing exons' (MEs), the proportion of annotated exons totally missed by the predictions (that is, there is no overlap by a predicted exon by at least 1 bp), and 'wrong exons' (WEs), the proportion of predicted exons not overlapping annotated exons by at least 1 bp. A subset of predicted exons falling in regions annotated as intergenic have been tested experimentally (see the section 'The experimental test of unannotated predictions' below for details). Nucleotide and exon level accuracy are calculated for the CDS evaluation and for the mRNA evaluation. Comparison of the results of these evaluation strategies highlights the differences for those programs that attempt to predict untranslated regions (UTRs) of genes.

The transcript and gene level accuracy measures are more stringent. We consider a transcript accurately predicted for the CDS evaluation if the beginning and end of translation are correctly annotated and each of the 5' and 3' splice sites for the coding exons are correct. Similarly, for the mRNA evaluation, a transcript is counted correct if all of the exons from the start of transcription to the end of transcription are correctly predicted. Thus, at the transcript level, sensitivity is the proportion of annotated transcripts that is correctly predicted, and specificity is the proportion of predicted transcripts that is correct. A gene is counted correct if at least one transcript in the locus is correct as defined above, and sensitivity and specificity are defined accordingly. Using these definitions, transcript accuracy is the most stringent measure for both the CDS evaluation and for the mRNA evaluation (Figure 4).

Figure 4

Gene transcript evaluation

Gene transcript evaluation. Computing sensitivity and specificity at transcript level: (a) complete transcript annotation; (b) incomplete transcript annotation. Transcripts marked with an asterisk are considered 'consistent with the annotation' and will be scored as correct.

The accuracy of the prediction methods must be considered in the context of the annotation, which contains a significant fraction of incomplete transcripts. In the case of an incomplete transcript, we made the distinction that if a prediction is completely consistent with the annotation, it will be counted correct. For example, if the annotation contains an incomplete transcript with three exons and a prediction method includes a transcript with these exons plus an additional exon, we consider the prediction to be completely consistent with the annotation and count it as a correct prediction. For the CDS evaluation, if the annotation contains a complete coding transcript, it must be predicted correctly and no additional exons are allowed (Figure 4).

Global results and trends

The evaluation statistics discussed above for the CDS evaluation are provided in Tables 4 and 5 and for the mRNA evaluation in Table 6, which only lists methods that predict full mRNA transcripts. Figures 5, 6, 7, 8 display the results for the CDS evaluation at the nucleotide, exon, transcript and gene levels. Values are given for programs in categories 1 to 4 (see previous section and Table 3), which constitute the bulk of the submitted predictions. The accuracies of the programs in other categories are often not strictly comparable and, therefore, not shown in these figures. They are, however, given in the Supplementary material 51. The top panel in Figures 5, 6, 7, 8 is a dotplot of sensitivity versus specificity, where each dot represents the performance of one program. The bottom panel includes a boxplot for each program displaying the average of sensitivity and specificity (that is, (Sn + Sp)/2) for the given program on each of 27 test sequences (see Materials and methods). Four test sequences (ENr112, ENr113, ENr311, ENr313) were removed from the original set of 31 because they did not contain any annotated protein coding genes and, therefore, sensitivity and specificity could not be computed for them. The dotplot intends to capture the global balance between sensitivity and specificity for each program, while the boxplots provide the dispersion of the accuracy of each program predictions across test sequences. At similar average accuracies, programs providing more consistent predictions across sequences may be preferable since their behavior can be better anticipated.

Table 4

CDS assessment: summary of accuracy measures for CDS features at the nucleotide and exon levels

Nucleotide

Exon

NSn

NSp

N CC

ESn

ESp

Category 1

AUGUSTUS-any

94.42%

82.43%

0.88

74.67%

76.76%

8.25%

16.29%

FGENESH++

91.09%

76.89%

0.83

75.18%

69.31%

9.73%

24.64%

JIGSAW

94.56%

92.19%

0.93

80.61%

89.33%

6.22%

7.78%

PAIRAGON-any

87.77%

92.78%

0.90

76.85%

88.91%

11.18%

6.82%

Category 2

AUGUSTUS-abinit

78.65%

75.29%

0.76

52.39%

62.93%

29.09%

24.82%

GENEMARK.hmm-A

78.43%

37.97%

0.53

50.58%

29.01%

27.86%

63.27%

GENEMARK.hmm-B

76.09%

62.94%

0.69

48.15%

47.25%

31.77%

40.68%

GENEZILLA

87.56%

50.93%

0.66

62.08%

50.25%

19.14%

41.93%

Category 3

ACEVIEW

90.94%

79.14%

0.84

85.75%

56.98%

4.38%

16.69%

AUGUSTUS-EST

92.62%

83.45%

0.88

74.10%

77.40%

9.01%

15.61%

ENSEMBL

90.18%

92.02%

0.91

77.53%

82.65%

9.99%

9.22%

EXOGEAN

84.18%

94.33%

0.89

79.34%

83.45%

9.88%

5.06%

EXONHUNTER

90.46%

59.67%

0.73

64.44%

41.77%

14.29%

50.94%

PAIRAGON+NSCAN_EST

87.56%

92.77%

0.90

76.63%

88.95%

11.51%

6.85%

Category 4

AUGUSTUS-dual

88.86%

80.15%

0.84

63.06%

69.14%

16.82%

19.60%

DOGFISH

64.81%

88.24%

0.74

53.11%

77.34%

32.67%

11.70%

MARS

84.25%

74.13%

0.78

65.56%

61.65%

20.26%

26.10%

NSCAN

85.38%

89.02%

0.87

67.66%

82.05%

17.11%

10.93%

SAGA

52.54%

81.39%

0.65

38.82%

50.73%

40.48%

27.85%

UCSC Tracks

ACEMBLY

96.43%

58.47%

0.74

84.66%

38.32%

2.71%

28.55%

CCDSgene

56.87%

99.52%

0.75

51.95%

97.75%

40.38%

0.27%

ECgene

96.36%

47.30%

0.66

86.22%

35.08%

2.64%

45.92%

ENSgene

91.39%

91.92%

0.92

77.71%

82.39%

9.80%

9.21%

GENEID

76.77%

76.48%

0.76

53.84%

61.08%

27.86%

27.26%

GENSCAN

84.17%

60.60%

0.71

58.65%

46.37%

19.50%

42.91%

KNOWNgene

89.10%

93.61%

0.91

78.11%

82.28%

10.27%

4.30%

MGCgene

44.06%

97.56%

0.65

42.95%

93.61%

49.28%

2.68%

REFgene

85.34%

98.50%

0.92

73.23%

94.67%

15.38%

1.22%

SGPgene

82.81%

82.20%

0.82

60.56%

65.16%

19.36%

22.85%

TWINSCAN

78.16%

84.59%

0.81

58.43%

73.11%

24.64%

16.30%

CC, correlation coefficient.

Table 5

CDS assessment at the transcript and gene levels

Transcript

Gene

TSn

TSp

GSn

GSp

Ratio CDS/UTR

Category 1

AUGUSTUS-any

22.65%

35.59%

47.97%

35.59%

100.00%

FGENESH++

36.21%

41.61%

69.93%

42.09%

78.25%

JIGSAW

34.05%

65.95%

72.64%

65.95%

100.00%

PAIRAGON-any

39.29%

60.34%

69.59%

61.32%

62.92%

Category 2

AUGUSTUS-abinit

11.09%

17.22%

24.32%

17.22%

100.00%

GENEMARK.hmm-A

6.93%

3.24%

15.20%

3.24%

100.00%

GENEMARK.hmm-B

7.70%

7.91%

16.89%

7.91%

100.00%

GENEZILLA

9.09%

8.84%

19.59%

8.84%

100.00%

Category 3

ACEVIEW

44.68%

19.31%

63.51%

48.65%

49.15%

AUGUSTUS-EST

22.50%

37.01%

47.64%

37.01%

100.00%

ENSEMBL

39.75%

54.64%

71.62%