We manually examined 33 external databases and identified 11 that had the depth and breadth appropriate for general disease annotation (Table 1). We chose the original 33 based on disease coverage and comprehensiveness and their amenability to automation via screen scraping or other methods. We also chose only those resources that could be queried by disease to return a gene list and associated attributes, such as supporting citations.

We designed an algorithm to query a given disease term in the 11 databases described in Table 1. The code was written in Java and Python and designed to form the union of the lists of genes returned from a disease query while recording whether a gene was present in or absent from each database. We also designed the code to run on a research cluster so that multiple requests could be run in batch on 6 available nodes. The length a single job was dependent on the size of the result set and on the response times of the 11 external resources. As a secondary step, the algorithm queried NCBI with the list of gene symbols to provide an enriched set of annotations including official gene symbol, Ensembl ID, gene ID, gene name, chromosome, location, symbol synonyms, and aliases (Table 2). If separate entries in the results were found to be synonyms, the algorithm collapsed the rows into a single entry under the official NCBI name taking the consensus of the results from all databases queried. The final results were formatted in a single tab-delimited text file for display/download on the Genotator website (example of pipeline provided in Figure 1).

In addition, we included a select group of high-value parameters to enrich the quality of the data with regards to clinical and experimental relevance of each gene. Specifically, we included from HuGE Navigator 7 and the Genetic Association Database 8 the manually assigned labels "Yes Association", "No Association", "Unknown Association" (where yes association indicates positive support for a gene's role in the disease phenotype) for each gene-article pair, p-values from genome-wide association studies, and the Gene Prospector score of association 9 . All key parameters were then compiled into an attribute vector for scoring and ranking of genes (described in the section immediately below). Finally, to obtain a list of publications supporting a gene-disease pair, we queried HuGE Navigator first with the disease of interest and then separately with the gene name (in this case the official gene symbol) and reported the intersecting publications in the final results.

The final step of the Genotator algorithm was the implementation of a scoring system to assign the strength of association per gene to the disease of interest. The Genotator score (GS) compiled information from the 11 databases using the following formula.

G S = G A D Y − G A D N + ϕ ( G P S )     + 1 / γ ( D B ) + 1 / κ ( R E F )

GAD_Y = Total number of "Yes" labeled associations for the gene and disease in the Genetic Association Database

GAD_N = Total number of "No" labeled associations for the gene and disease in the Genetic Association Database

GPS = Gene Prospector's score of gene-disease association

DB = Number of total databases (out of 11) that the gene appeared in

REF = Number of total references for the given gene

The constants ϕ , γ , and κ were used to weight the contributions of the GPS, DB, and REF parameters. We elected to set these values at 100, 10, and 5, respectively, as these numbers provided the best overall performance. However, we designed the Genotator software to enable users the ability to alter these parameters should alternative weighting schemes be desired. In addition, Genotator reports the complete set of attributes together with the GS score, thus enabling users the flexibility to retain the original scoring scheme or to define an alternative strategy. We devised the GS to serve as guide for gene prioritization that best captures the history of biomedical research findings while also ensuring adjustability of the threshold for inclusion/exclusion of genes.

We elected to use three trusted, manually updated, and disease-specific databases to provide (1) a list of vetted candidate genes that have clear association to the disease based on published research (linkage studies, genome-wide association studies, etc.), and (2) an independent ranking of the strength of association between the genes and the disease of interest against which to compare ranked results from Genotator. The three databases, SFARI Gene, PDGene, and AlzGene target Autism Spectrum Disorder (ASD), Parkinson Disease (PD), and Alzheimer's Disease (AD), respectively. SFARI gene is a leading repository of genetic information for ASD that is updated with high frequency by manual curators. PDGene is one of the most trusted web resources for genes associated with PD and has previously been used to validate the Gene Prospector scoring algorithm in Yu et al. 9 . AlzGene 6 has a similar structure to PDGene, and is also widely regarded as an authoritative source of information on the genes associated with AD. To generate a metric for ranking the genes reported by SFARI, we subtracted the total number of negative from the total number of positive associations that were directly reported in the database. PDGene and AlzGene provided a ranking of the top gene candidates in section of their websites called "TopGenes". At the time of writing, PDGene listed 32 and AlzGene listed 40 top genes. In addition, both resources assigned a label of "strong", "moderate", or "weak" to each top gene candidate based upon amount of evidence, extent of replication and protection from bias 10 . By utilizing these resources as a baseline for comparison, we could determine how well the automated procedures employed by Genotator compared to standards of manual annotation, both in terms of coverage of published research as well as in terms of prioritization of gene candidates.

We designed a web resource for open access to the Genotator pipeline, available at http://genotator.hms.harvard.edu. The site has two primary components, one for initiating a query through a request form, and another for storing results under a disorder name (Figure 2). Results are stored persistently, together with the creation date. A user may request to update the results for a specific disorder if the results are likely to be outdated based on the last creation date. The request form requires a disorder name, following the standard vocabulary of the Unified Medical Language System, and an active email address. An automatic email is sent to the user when the pipeline completes. The results are then displayed in the "Disorders" tab and saved until updated. Each result set can be displayed in the browser or downloaded in full as a .txt file. The output contains a presence/absence matrix of the databases in which the gene appears; this gives the user the ability to select among the 11 databases and also to rank/reorder the results using criteria other than the Genotator score.

When using "Autism" as the query term in the Genotator disease search, 663 genes were returned. Six of the top 10 genes returned were listed among the most promising autism candidates in a recent review (Table 4) 11 , demonstrating that the top ranked genes by Genotator match recent advances in our understanding of autism genetics. Using our method to rank the 197 SFARI genes, we compared the ordered lists from SFARI and Genotator. Genotator found a total of 77% (152 out of the 197 SFARI 10 genes). All of the top 10 SFARI genes were included in the Genotator set, as well as 19 out of the top 20, 29 out of the top 30, 35 out of the top 40, and 41 out of the top 50.

As in Yu et al. 2008, we used the well-known PDGene database as a source for validation of the accuracy and coverage of Genotator. The PDGene database provided a relative ranking (based on HuGENet/Venice grades 10 ) that enabled us to determine the extent of overlap with Genotator, both in terms of coverage and strength of association to the disease. Twenty-one of the 32 PDGene "Top Results" genes were in the top 10th percentile of the results returned by Genotator, and in total 58% (312/540 PDGene genes) were found to be in common (Table 5). In addition, 6 of the top 10 Genotator genes were listed among the most promising PD gene candidates in a recent review (Table 5) 12 , indicating that Genotator appropriately prioritized leading research in the field.

As a final use case, we focused on AD and the well-known AlzGene database. Similar to the PDGene database, AlzGene provided a relative ranking (based on HuGENet/Venice grades 10 ) that enabled us to determine the extent of overlap with Genotator in terms of coverage and strength of association to the disease. Of the three disease use cases, Genotator appeared to perform best with Alzheimer Disease. Thirty-four of the 40 AlzGene "Top Results" genes were in the top 10th percentile of the results returned by Genotator, and in total 74% (486/660) of the candidate genes reported in AlzGene were found by Genotator (Table 6).