<?xml version='1.0'?>
<!DOCTYPE art SYSTEM 'http://www.biomedcentral.com/xml/article.dtd'>
<art>
   <ui>1471-2105-4-12</ui>
   <ji>1471-2105</ji>
   <fm>
      <dochead>Methodology article</dochead>
      <bibl>
         <title>
            <p>Significance analysis of lexical bias in microarray data</p>
         </title>
         <aug>
            <au id="A1" ca="yes">
               <snm>Kim</snm>
               <mi>C</mi>
               <fnm>Charles</fnm>
               <insr iid="I1"/>
               <email>cckim@stanford.edu</email>
            </au>
            <au id="A2">
               <snm>Falkow</snm>
               <fnm>Stanley</fnm>
               <insr iid="I1"/>
               <email>falkow@stanford.edu</email>
            </au>
         </aug>
         <insg>
            <ins id="I1">
               <p>Microbiology and Immunology, Stanford University Medical Center, Stanford, CA, 94305, USA</p>
            </ins>
         </insg>
         <source>BMC Bioinformatics</source>
         <issn>1471-2105</issn>
         <pubdate>2003</pubdate>
         <volume>4</volume>
         <issue>1</issue>
         <fpage>12</fpage>
         <url>http://www.biomedcentral.com/1471-2105/4/12</url>
         <xrefbib>
            <pubidlist>
               <pubid idtype="doi">10.1186/1471-2105-4-12</pubid>
               <pubid idtype="pmpid">12697067</pubid>
            </pubidlist>
         </xrefbib>
      </bibl>
      <history>
         <rec>
            <date>
               <day>8</day>
               <month>1</month>
               <year>2003</year>
            </date>
         </rec>
         <acc>
            <date>
               <day>3</day>
               <month>4</month>
               <year>2003</year>
            </date>
         </acc>
         <pub>
            <date>
               <day>3</day>
               <month>4</month>
               <year>2003</year>
            </date>
         </pub>
      </history>
      <cpyrt>
         <year>2003</year>
         <collab>Kim and Falkow; licensee BioMed Central Ltd. This is an Open Access article: verbatim copying and redistribution of this article are permitted in all media for any purpose, provided this notice is preserved along with the article's original URL.</collab>
      </cpyrt>
      <abs>
         <sec>
            <st>
               <p>Abstract</p>
            </st>
            <sec>
               <st>
                  <p>Background</p>
               </st>
               <p>Genes that are determined to be significantly differentially regulated in microarray analyses often appear to have functional commonalities, such as being components of the same biochemical pathway. This results in certain words being under- or overrepresented in the list of genes. Distinguishing between biologically meaningful trends and artifacts of annotation and analysis procedures is of the utmost importance, as only true biological trends are of interest for further experimentation. A number of sophisticated methods for identification of significant lexical trends are currently available, but these methods are generally too cumbersome for practical use by most microarray users.</p>
            </sec>
            <sec>
               <st>
                  <p>Results</p>
               </st>
               <p>We have developed a tool, LACK, for calculating the statistical significance of apparent lexical bias in microarray datasets. The frequency of a user-specified list of search terms in a list of genes which are differentially regulated is assessed for statistical significance by comparison to randomly generated datasets. The simplicity of the input files and user interface targets the average microarray user who wishes to have a statistical measure of apparent lexical trends in analyzed datasets without the need for bioinformatics skills. The software is available as Perl source or a Windows executable.</p>
            </sec>
            <sec>
               <st>
                  <p>Conclusion</p>
               </st>
               <p>We have used LACK in our laboratory to generate biological hypotheses based on our microarray data. We demonstrate the program's utility using an example in which we confirm significant upregulation of SPI-2 pathogenicity island of <it>Salmonella enterica </it>serovar Typhimurium by the cation chelator dipyridyl.</p>
            </sec>
         </sec>
      </abs>
   </fm>
   <meta>
      <classifications>
         <classification type="bmc" subtype="user_supplied_xml" id="endnote"/>
      </classifications>
   </meta>
   <bdy>
      <sec>
         <st>
            <p>Background</p>
         </st>
         <p>Many methods are available for the analysis of expression data from microarrays <abbrgrp><abbr bid="B1">1</abbr><abbr bid="B2">2</abbr><abbr bid="B3">3</abbr></abbrgrp>. Most software falls into one of two categories: programs which generate a list of significantly differentially regulated genes and programs which perform an analysis based on such a significant genes list (SGL). The first category includes statistical methods (e.g. SAM <abbrgrp><abbr bid="B4">4</abbr></abbrgrp>) and pattern-grouping methods (e.g. Cluster <abbrgrp><abbr bid="B5">5</abbr></abbrgrp>), the end result of either type of analysis being a list of genes which appear to be significantly differentially regulated. The second category of software includes a variety of approaches which search for relationships between the members of the SGL, such as methods which identify common regulatory motifs <abbrgrp><abbr bid="B6">6</abbr></abbrgrp> or establish significance based on genetic position <abbrgrp><abbr bid="B7">7</abbr></abbrgrp>.</p>
         <p>In browsing a SGL, lexical trends are frequently observed as genes involved in similar metabolic functions are often coordinately regulated. Identification of such lexical trends is critical for the generation of hypotheses for further experimentation, but the large size of a typical microarray dataset makes objective evaluation by eye difficult, if not impossible. Additionally, the genetic content of the organism, the style of the genome annotation, the features selected for representation on the microarray, and experimental design all contain biases which are ultimately reflected in the SGL, which further confounds objective analysis. A number of publications have addressed some of these issues by designing lexical analyses based on functional classifications <abbrgrp><abbr bid="B8">8</abbr></abbrgrp>, Medline MeSH terms <abbrgrp><abbr bid="B9">9</abbr></abbrgrp>, or Medline titles and abstracts <abbrgrp><abbr bid="B10">10</abbr><abbr bid="B11">11</abbr><abbr bid="B12">12</abbr></abbrgrp> that are associated with the genes in a SGL. Each of these approaches draw upon external information to overlay biological relationships onto the expression data, and the significance of these relationships can then be analyzed using standard statistical tools.</p>
         <p>The described methods have all been used to identify previously undetected relationships of possible biological significance in published microarray datasets. However, each approach has drawbacks that limit their use as a generalized analysis tool for the average microarray user. Methods based on functional classifications require that the genes be categorized prior to the analysis. This is often not performed by the sequencing group and too laborious to be practical for most microarray end users. Medline-based methods cannot include genes with putative functions in their analyses, as there is no associated literature for such genes. As over 50% of the genes in most genomes are still of putative function, abstract-based approaches exclude a large proportion of the expression data from the analysis. All of the described approaches are also confounded by ambiguity and variability in the annotations used by different researchers (e.g. genes with multiple names). Most importantly, these methods require a significant degree of computer savvy to set up and execute, which makes use impractical for many microarray users. We developed a program, LACK, which performs a simple but flexible lexical analysis of microarray data that circumvents many of the described problems.</p>
      </sec>
      <sec>
         <st>
            <p>Results and Discussion</p>
         </st>
         <sec>
            <st>
               <p>Observed Frequency of Search Terms in the SGL</p>
            </st>
            <p>The software requires three input files. The first file contains the SGL. This is usually the output of a clustering or statistical analysis which has been conducted to identify differentially regulated genes. The second file is the full, unanalyzed dataset. This dataset should be identical to the dataset used for the SGL generation to avoid introduction of any bias from the pre-analysis steps (array design, hybridization, feature identification, data filtering, etc.). The third file is a user-specified set of search terms to be used for the lexical analysis.</p>
            <p>The software first tallies the number of genes in the SGL containing any of the search terms. The gene is counted as a match if any of the search terms are found within the annotation. The total tally of matches within the SGL represents the observed frequency of the search terms in the analyzed data. The software then identifies the frequencies at which the search terms would be found at random in datasets of identical size to the SGL. Two approaches for generating and analyzing the random datasets are available: binomial and Poisson. The choice is dependent on the size of the datasets being used due to computational limitations (described below).</p>
         </sec>
         <sec>
            <st>
               <p>Binomial Statistics</p>
            </st>
            <p>The default statistical method employed is the binomial distribution to model the data. For each member of the test sample, the binary criterion (contains a search term / does not contain any search terms) is applied. Furthermore, the population from which the test sample (SGL) is chosen is already known (the full dataset), so the precise frequencies of genes containing a search term (<it>p </it>= number of genes containing a match / number of genes) or not containing any of the search terms (<it>q </it>= 1 - <it>p</it>) can be calculated. For a given population, we can calculate the probability of the observed number of matches (or fewer) in the SGL (<it>k</it>) occurring in a random sample of size <it>N </it>by the cumulative binomial probability function:</p>
            <p>
               <graphic file="1471-2105-4-12-i1.gif"/>
            </p>
            <p>By using the binomial calculation, there is no need for random sampling, which has the advantages of higher precision in the <it>p</it>-values and reduced computation time. The probability of obtaining <it>j </it>matches, <it>P</it>(<it>j</it>), is calculated for every value of <it>j </it>which generates <it>P</it>(<it>j</it>) = 0.00001 and provided in the output in order to generate enough values for plotting both tails of a histogram.</p>
         </sec>
         <sec>
            <st>
               <p>Poisson Analysis</p>
            </st>
            <p>Because microarray datasets are often very large, the factorial computations in the binomial coefficient can easily exceed the capabilities of modern desktop computers. We therefore provide an option to perform Poisson statistics, which does not require very large integers. Random samples (RSs) that are identical in size to the SGL are repeatedly chosen from the full dataset with replacement, and the number of genes containing any of the search terms is tallied for each RS. The cumulative Poisson probability is calculated for the observed number of matches in the analyzed dataset as compared to the distribution of matches in the RSs:</p>
            <p>
               <graphic file="1471-2105-4-12-i2.gif"/>
            </p>
            <p>where <it>k </it>is the number of matches in the SGL and &#956; is the mean number of matches for all RSs.</p>
            <p>We have observed that 100 RSs are generally sufficient to accurately mimic the <it>p</it>-values generated by binomial analysis (Table <tblr tid="T1">1</tblr>). However, using 1000 RSs requires only slightly more computational time, which is easily outweighed by the benefit of a more accurate <it>p</it>-value. It is also noteworthy that for more significant numbers of matches, there is less variation in the <it>p</it>-values. Therefore, we recommend performing multiple trials when using Poisson statistics, especially in cases where the <it>p</it>-value borders on the selected significance threshold.</p>
            <tbl id="T1">
               <title>
                  <p>Table 1</p>
               </title>
               <caption>
                  <p>Variation in cumulative Poisson probabilities in response to number of random samples and SGL matches</p>
               </caption>
               <tblbdy cols="7">
                  <r>
                     <c>
                        <p/>
                     </c>
                     <c cspan="6" ca="center">
                        <p>
                           <b>Matches in SGL</b>
                        </p>
                     </c>
                  </r>
                  <r>
                     <c>
                        <p/>
                     </c>
                     <c cspan="6">
                        <hr/>
                     </c>
                  </r>
                  <r>
                     <c ca="center">
                        <p>
                           <b>RSs</b>
                        </p>
                     </c>
                     <c ca="center">
                        <p>
                           <b>8</b>
                        </p>
                     </c>
                     <c ca="center">
                        <p>
                           <b>7</b>
                        </p>
                     </c>
                     <c ca="center">
                        <p>
                           <b>6</b>
                        </p>
                     </c>
                     <c ca="center">
                        <p>
                           <b>5</b>
                        </p>
                     </c>
                     <c ca="center">
                        <p>
                           <b>4</b>
                        </p>
                     </c>
                     <c ca="center">
                        <p>
                           <b>3</b>
                        </p>
                     </c>
                  </r>
                  <r>
                     <c cspan="7">
                        <hr/>
                     </c>
                  </r>
                  <r>
                     <c ca="center">
                        <p>
                           <b>10</b>
                        </p>
                     </c>
                     <c ca="center">
                        <p>0.993 &#177; 0.014</p>
                     </c>
                     <c ca="center">
                        <p>0.981 &#177; 0.045</p>
                     </c>
                     <c ca="center">
                        <p>0.957 &#177; 0.057</p>
                     </c>
                     <c ca="center">
                        <p>0.898 &#177; 0.126</p>
                     </c>
                     <c ca="center">
                        <p>0.799 &#177; 0.167</p>
                     </c>
                     <c ca="center">
                        <p>0.626 &#177; 0.240</p>
                     </c>
                  </r>
                  <r>
                     <c ca="center">
                        <p>
                           <b>100</b>
                        </p>
                     </c>
                     <c ca="center">
                        <p>0.996 &#177; 0.003</p>
                     </c>
                     <c ca="center">
                        <p>0.987 &#177; 0.008</p>
                     </c>
                     <c ca="center">
                        <p>0.963 &#177; 0.017</p>
                     </c>
                     <c ca="center">
                        <p>0.914 &#177; 0.033</p>
                     </c>
                     <c ca="center">
                        <p>0.812 &#177; 0.059</p>
                     </c>
                     <c ca="center">
                        <p>0.643 &#177; 0.072</p>
                     </c>
                  </r>
                  <r>
                     <c ca="center">
                        <p>
                           <b>1000</b>
                        </p>
                     </c>
                     <c ca="center">
                        <p>0.996 &#177; 0.001</p>
                     </c>
                     <c ca="center">
                        <p>0.987 &#177; 0.003</p>
                     </c>
                     <c ca="center">
                        <p>0.964 &#177; 0.005</p>
                     </c>
                     <c ca="center">
                        <p>0.913 &#177; 0.011</p>
                     </c>
                     <c ca="center">
                        <p>0.810 &#177; 0.017</p>
                     </c>
                     <c ca="center">
                        <p>0.647 &#177; 0.021</p>
                     </c>
                  </r>
               </tblbdy>
               <tblfn>
                  <p>Poisson analysis was performed on the described dataset 100 times using 10, 100, or 1000 random samples. The mean of the cumulative probabilities is reported with two standard deviations. The original SGL contained 8 matches (first column); the additional columns were generated using a synthetic SGL modified to contain the specified number of matches.</p>
               </tblfn>
            </tbl>
            <p>Additionally, in Poisson output mode, a raw output option can be selected that includes the tally data in the output file for analysis by other statistical methods.</p>
         </sec>
         <sec>
            <st>
               <p>Validation of LACK</p>
            </st>
            <p>We have successfully used LACK to generate and test hypotheses generated from our microarray data. Using our <it>Salmonella enterica </it>serovar Typhimurium LT2 microarray <abbrgrp><abbr bid="B13">13</abbr></abbrgrp>, we observed that members of <it>Salmonella </it>Pathogenicity Island 2 (SPI-2) <abbrgrp><abbr bid="B14">14</abbr><abbr bid="B15">15</abbr></abbrgrp> appeared 8 times in the top 256 genes significantly upregulated by the ferrous iron chelator dipyridyl. Because there are 34 genes predicted to be within SPI-2 <abbrgrp><abbr bid="B14">14</abbr><abbr bid="B15">15</abbr></abbrgrp>, it seemed possible that this number of matches might appear at random. We constructed a list of the 34 genes in SPI-2 and performed LACK analysis on the dataset. The binomial distribution of the calculated expected frequency of matches and the probability values show that the overrepresentation of SPI-2 in the dataset is unlikely to be random (Figure <figr fid="F1">1</figr>). We have confirmed that SPI-2 is indeed induced by dipyridyl treatment using a SPI-2 promoter (<it>ssaG</it>) fused to GFP (data not shown). During the course of this work, an independent report also confirmed that SPI-2 is indeed upregulated in response to cation starvation <abbrgrp><abbr bid="B16">16</abbr></abbrgrp>.</p>
            <fig id="F1">
               <title>
                  <p>Figure 1</p>
               </title>
               <caption>
                  <p>Binomial distribution of SPI-2 genes in a dataset</p>
               </caption>
               <text>
                  <p><b>Binomial distribution of SPI-2 genes in a dataset </b>The total filtered dataset consisted on 4290 unique elements. An SGL of 256 genes was generated using SAM and analyzed for 34 members of SPI-2. The arrow indicates the number of matches in the SGL, with <it>P</it>(<it>x </it>> 8) = 0.004. The binomial analysis required 5 seconds; Poisson analysis of the same datasets required 7 seconds. A 21,450 element dataset created by replicating the 4290 element dataset 5 times required 8 seconds for binomial analysis. The files used for this analysis are available at the LACK website or as supplementary data.</p>
               </text>
               <graphic file="1471-2105-4-12-1"/>
            </fig>
         </sec>
      </sec>
      <sec>
         <st>
            <p>Conclusions</p>
         </st>
         <p>We have demonstrated one example of how we have employed LACK for statistical analysis of lexical trends. In analyzing our microarray data, we realized that few simple tools are available for assessing significance of observed trends. While there are a number of other lexical analysis methods available, they employ different goals (<it>i.e. </it>de novo identification of lexical trends) than LACK (significance analysis of specific lexical trends). The primary advantages of LACK over other lexical analysis methods for the typical microarray user are the simplicity of use and the flexibility in searchable concepts. The inputs are text files that are generated as part of a typical microarray analysis, and the output is provided as tab-delimited text; no external information is required apart from the microarray annotation. The inherent flexibility in user-specified search terms makes LACK potentially useful for many microarray users who are interested in determining whether or not specific trends are significant in their data, rather than performing a global search of all possible trends. In addition, this flexibility makes lexical analysis possible for datasets that are difficult to analyze by other lexical methods due to inconsistencies in annotation nomenclature, as the user can specify terms to encompass multiple naming conventions.</p>
         <p>It is important to note that with this flexibility comes an additional point at which bias can be introduced: careless selection of search terms can lead to false determination of significance. For example, search terms should not be chosen only from the SGL itself, but rather from the full dataset. To demonstrate such an inappropriate example, we selected a search list based on metabolites found in the top 20 genes of the SGL described above. While there is no clear relationship between the search list terms (glucosamine, methionine sulfoxide, citrate, 2-aminoethylphosphonate, ethanolamine, UDP-glucose, cysteine), LACK indicates a high level of significance in these terms being over-represented in the SGL (<it>p </it>&lt; 0.0005), illustrating the danger of choosing terms directly from the SGL.</p>
         <p>One possible approach that avoids this problem is to select a wordlist prior to generation of the SGL. This has the obvious limitation that it cannot be implemented for exploratory microarray experiments, only for specific hypothesis testing. Another approach is to generate the word list in a blinded fashion; for example, a researcher who did not generate the SGL could select search terms from the full dataset. We have also made a tool available, ALACK (automated LACK), which selects single search terms which are over-represented in the SGL in an automated, unbiased fashion (although multiple search terms are not supported in ALACK). While the benefits of LACK far outweigh the potential abuses, it is clear that care must be taken to avoid over-interpretation of significance.</p>
         <p>Analysis of the significance of trends in expression data is critical for the generation of further hypotheses, as variation in the arrays and in annotations can confound visual assessment of significance. We have described a method for assessing the statistical significance of lexical trends in microarray data. Our approach is similar to, but distinct from, other lexical analysis methods <abbrgrp><abbr bid="B3">3</abbr><abbr bid="B8">8</abbr><abbr bid="B9">9</abbr><abbr bid="B10">10</abbr><abbr bid="B11">11</abbr></abbrgrp>, in that our method does not incorporate external information. Rather, LACK is designed to analyze the SGL generated from another analysis within the context of the microarray from which the data was generated. We expect that LACK can serve as a useful adjunct to other microarray analysis methods, including other global lexical trend identification methods.</p>
      </sec>
      <sec>
         <st>
            <p>Methods</p>
         </st>
         <sec>
            <st>
               <p>LACK Programming</p>
            </st>
            <p>LACK was written and tested using ActiveState Perl Build 631 on an Athlon 1.4 GHz system running Windows 2000 Pro. The graphical user interface uses Perl/Tk. The source has also been successfully tested on RedHat Linux 7.3.</p>
         </sec>
         <sec>
            <st>
               <p>Microarray Experiments</p>
            </st>
            <p>Our Salmonella enterica serovar Typhimurium microarray is described elsewhere <abbrgrp><abbr bid="B13">13</abbr></abbrgrp>. Typhimurium SL1344 was grown in unagitated cultures of 100 ml M9 minimal medium in 125 ml Erlenmeyer flasks to an OD<sub>600 </sub>of 0.2. The culture was divided in half and transferred to 50 ml Erlenmeyer flasks. 2,2'-dipyridyl was added to 500 &#956;M to one of the half cultures. Aliquots (10 ml) were removed at 10 min, 30 min, and 60 min post-addition and directly added to 1/10 volume of 95% ethanol / 5% phenol stop-solution. RNA was prepared by a modified Qiagen RNeasy prep protocol <url>http://falkow.stanford.edu</url> and hybridized using standard reverse transcription and cDNA-labeling procedures <url>http://www.microarrays.org</url>.</p>
         </sec>
      </sec>
      <sec>
         <st>
            <p>Authors' contributions</p>
         </st>
         <p>CK performed programming of LACK, microarray experiments, and preparation of the manuscript. SF provided funding and supervision for the experiments described herein. Both authors have read and approved the final manuscript.</p>
         <suppl id="S1">
            <title>
               <p>Additional File 1</p>
            </title>
            <text>
               <p>LACK Windows Executable. Additional file descriptions text (including details of how to view the file, if it is in a non-standard format).</p>
            </text>
            <file name="1471-2105-4-12-S1.zip">
               <p>Click here for file</p>
            </file>
         </suppl>
         <suppl id="S2">
            <title>
               <p>Additional File 2</p>
            </title>
            <text>
               <p>LACK Perl source code. Additional file descriptions text (including details of how to view the file, if it is in a non-standard format).</p>
            </text>
            <file name="1471-2105-4-12-S2.pl">
               <p>Click here for file</p>
            </file>
         </suppl>
         <suppl id="S3">
            <title>
               <p>Additional File 3</p>
            </title>
            <text>
               <p>LACK Manual. Additional file descriptions text (including details of how to view the file, if it is in a non-standard format).</p>
            </text>
            <file name="1471-2105-4-12-S3.pdf">
               <p>Click here for file</p>
            </file>
         </suppl>
         <suppl id="S4">
            <title>
               <p>Additional File 4</p>
            </title>
            <text>
               <p>Sample Data. Three sample test files, identical to those used for the described analysis, are available. The files are full.txt (full dataset), analyzed.txt (analyzed dataset, SGL), and words.txt (34 members of SPI-2).</p>
            </text>
            <file name="1471-2105-4-12-S4.zip">
               <p>Click here for file</p>
            </file>
         </suppl>
      </sec>
   </bdy>
   <bm>
      <ack>
         <sec>
            <st>
               <p>Acknowledgements</p>
            </st>
            <p>We thank D. Scott Merrell for suggesting improvements to the algorithm and Kaman Chan for reading the manuscript. CK thanks the Howard Hughes Medical Institute and the Stanford Graduate Fellowship Program for funding. This work was funded in part by the Stanford Digestive Disease Center grant DK56339.</p>
         </sec>
      </ack>
      <refgrp>
         <bibl id="B1">
            <title>
               <p>Computational analysis of microarray data</p>
            </title>
            <aug>
               <au>
                  <snm>Quackenbush</snm>
                  <fnm>J</fnm>
               </au>
            </aug>
            <source>Nat Rev Genet</source>
            <pubdate>2001</pubdate>
            <volume>2</volume>
            <fpage>418</fpage>
            <lpage>427</lpage>
            <xrefbib>
               <pubidlist>
                  <pubid idtype="pmpid" link="fulltext">11389458</pubid>
                  <pubid idtype="doi">10.1038/35076576</pubid>
                  <pubid idtype="pmcid">33173</pubid>
               </pubidlist>
            </xrefbib>
         </bibl>
         <bibl id="B2">
            <title>
               <p>Practical approaches to analyzing results of microarray experiments</p>
            </title>
            <aug>
               <au>
                  <snm>Kaminski</snm>
                  <fnm>N</fnm>
               </au>
               <au>
                  <snm>Friedman</snm>
                  <fnm>N</fnm>
               </au>
            </aug>
            <source>Am J Respir Cell Mol Biol</source>
            <pubdate>2002</pubdate>
            <volume>27</volume>
            <fpage>125</fpage>
            <lpage>132</lpage>
            <xrefbib>
               <pubidlist>
                  <pubid idtype="pmcid">24541</pubid>
                  <pubid idtype="pmpid" link="fulltext">12151303</pubid>
               </pubidlist>
            </xrefbib>
         </bibl>
         <bibl id="B3">
            <title>
               <p>Whole-genome expression analysis: challenges beyond clustering</p>
            </title>
            <aug>
               <au>
                  <snm>Altman</snm>
                  <fnm>RB</fnm>
               </au>
               <au>
                  <snm>Raychaudhuri</snm>
                  <fnm>S</fnm>
               </au>
            </aug>
            <source>Curr Opin Struct Biol</source>
            <pubdate>2001</pubdate>
            <volume>11</volume>
            <fpage>340</fpage>
            <lpage>347</lpage>
            <xrefbib>
               <pubidlist>
                  <pubid idtype="pmpid" link="fulltext">11406385</pubid>
                  <pubid idtype="doi">10.1016/S0959-440X(00)00212-8</pubid>
                  <pubid idtype="pmcid">18977</pubid>
               </pubidlist>
            </xrefbib>
         </bibl>
         <bibl id="B4">
            <title>
               <p>Significance analysis of microarrays applied to the ionizing radiation response</p>
            </title>
            <aug>
               <au>
                  <snm>Tusher</snm>
                  <fnm>VG</fnm>
               </au>
               <au>
                  <snm>Tibshirani</snm>
                  <fnm>R</fnm>
               </au>
               <au>
                  <snm>Chu</snm>
                  <fnm>G</fnm>
               </au>
            </aug>
            <source>Proc Natl Acad Sci U S A</source>
            <pubdate>2001</pubdate>
            <volume>98</volume>
            <fpage>5116</fpage>
            <lpage>5121</lpage>
            <xrefbib>
               <pubidlist>
                  <pubid idtype="pmpid" link="fulltext">11309499</pubid>
                  <pubid idtype="doi">10.1073/pnas.091062498</pubid>
                  <pubid idtype="pmcid">33173</pubid>
               </pubidlist>
            </xrefbib>
         </bibl>
         <bibl id="B5">
            <title>
               <p>Cluster analysis and display of genome-wide expression patterns</p>
            </title>
            <aug>
               <au>
                  <snm>Eisen</snm>
                  <fnm>MB</fnm>
               </au>
               <au>
                  <snm>Spellman</snm>
                  <fnm>PT</fnm>
               </au>
               <au>
                  <snm>Brown</snm>
                  <fnm>PO</fnm>
               </au>
               <au>
                  <snm>Botstein</snm>
                  <fnm>D</fnm>
               </au>
            </aug>
            <source>Proc Natl Acad Sci U S A</source>
            <pubdate>1998</pubdate>
            <volume>95</volume>
            <fpage>14863</fpage>
            <lpage>14868</lpage>
            <xrefbib>
               <pubidlist>
                  <pubid idtype="pmpid" link="fulltext">9843981</pubid>
                  <pubid idtype="doi">10.1073/pnas.95.25.14863</pubid>
                  <pubid idtype="pmcid">24541</pubid>
               </pubidlist>
            </xrefbib>
         </bibl>
         <bibl id="B6">
            <title>
               <p>BioProspector: discovering conserved DNA motifs in upstream regulatory regions of co-expressed genes</p>
            </title>
            <aug>
               <au>
                  <snm>Liu</snm>
                  <fnm>X</fnm>
               </au>
               <au>
                  <snm>Brutlag</snm>
                  <fnm>DL</fnm>
               </au>
               <au>
                  <snm>Liu</snm>
                  <fnm>JS</fnm>
               </au>
            </aug>
            <source>Pac Symp Biocomput</source>
            <pubdate>2001</pubdate>
            <fpage>127</fpage>
            <lpage>138</lpage>
            <xrefbib>
               <pubid idtype="pmpid">11262934</pubid>
            </xrefbib>
         </bibl>
         <bibl id="B7">
            <title>
               <p>Nitrogen regulatory protein C-controlled genes of Escherichia coli: scavenging as a defense against nitrogen limitation</p>
            </title>
            <aug>
               <au>
                  <snm>Zimmer</snm>
                  <fnm>DP</fnm>
               </au>
               <au>
                  <snm>Soupene</snm>
                  <fnm>E</fnm>
               </au>
               <au>
                  <snm>Lee</snm>
                  <fnm>HL</fnm>
               </au>
               <au>
                  <snm>Wendisch</snm>
                  <fnm>VF</fnm>
               </au>
               <au>
                  <snm>Khodursky</snm>
                  <fnm>AB</fnm>
               </au>
               <au>
                  <snm>Peter</snm>
                  <fnm>BJ</fnm>
               </au>
               <au>
                  <snm>Bender</snm>
                  <fnm>RA</fnm>
               </au>
               <au>
                  <snm>Kustu</snm>
                  <fnm>S</fnm>
               </au>
            </aug>
            <source>Proc Natl Acad Sci U S A</source>
            <pubdate>2000</pubdate>
            <volume>97</volume>
            <fpage>14674</fpage>
            <lpage>14679</lpage>
            <xrefbib>
               <pubidlist>
                  <pubid idtype="pmpid" link="fulltext">11121068</pubid>
                  <pubid idtype="doi">10.1073/pnas.97.26.14674</pubid>
                  <pubid idtype="pmcid">18977</pubid>
               </pubidlist>
            </xrefbib>
         </bibl>
         <bibl id="B8">
            <title>
               <p>Systematic determination of genetic network architecture</p>
            </title>
            <aug>
               <au>
                  <snm>Tavazoie</snm>
                  <fnm>S</fnm>
               </au>
               <au>
                  <snm>Hughes</snm>
                  <fnm>JD</fnm>
               </au>
               <au>
                  <snm>Campbell</snm>
                  <fnm>MJ</fnm>
               </au>
               <au>
                  <snm>Cho</snm>
                  <fnm>RJ</fnm>
               </au>
               <au>
                  <snm>Church</snm>
                  <fnm>GM</fnm>
               </au>
            </aug>
            <source>Nat Genet</source>
            <pubdate>1999</pubdate>
            <volume>22</volume>
            <fpage>281</fpage>
            <lpage>285</lpage>
            <xrefbib>
               <pubidlist>
                  <pubid idtype="pmpid" link="fulltext">10391217</pubid>
                  <pubid idtype="doi">10.1038/10343</pubid>
               </pubidlist>
            </xrefbib>
         </bibl>
         <bibl id="B9">
            <title>
               <p>Use of keyword hierarchies to interpret gene expression patterns</p>
            </title>
            <aug>
               <au>
                  <snm>Masys</snm>
                  <fnm>DR</fnm>
               </au>
               <au>
                  <snm>Welsh</snm>
                  <fnm>JB</fnm>
               </au>
               <au>
                  <snm>Lynn Fink</snm>
                  <fnm>J</fnm>
               </au>
               <au>
                  <snm>Gribskov</snm>
                  <fnm>M</fnm>
               </au>
               <au>
                  <snm>Klacansky</snm>
                  <fnm>I</fnm>
               </au>
               <au>
                  <snm>Corbeil</snm>
                  <fnm>J</fnm>
               </au>
            </aug>
            <source>Bioinformatics</source>
            <pubdate>2001</pubdate>
            <volume>17</volume>
            <fpage>319</fpage>
            <lpage>326</lpage>
            <xrefbib>
               <pubidlist>
                  <pubid idtype="pmpid" link="fulltext">11301300</pubid>
                  <pubid idtype="doi">10.1093/bioinformatics/17.4.319</pubid>
               </pubidlist>
            </xrefbib>
         </bibl>
         <bibl id="B10">
            <title>
               <p>Mining functional information associated with expression arrays</p>
            </title>
            <aug>
               <au>
                  <snm>Blaschke</snm>
                  <fnm>C</fnm>
               </au>
               <au>
                  <snm>Oliveros</snm>
                  <fnm>JC</fnm>
               </au>
               <au>
                  <snm>Valencia</snm>
                  <fnm>A</fnm>
               </au>
            </aug>
            <source>Funct Integr Genomics</source>
            <pubdate>2001</pubdate>
            <volume>1</volume>
            <fpage>256</fpage>
            <lpage>268</lpage>
            <xrefbib>
               <pubidlist>
                  <pubid idtype="pmpid" link="fulltext">11793245</pubid>
                  <pubid idtype="doi">10.1007/s101420000036</pubid>
               </pubidlist>
            </xrefbib>
         </bibl>
         <bibl id="B11">
            <title>
               <p>A literature network of human genes for high-throughput analysis of gene expression</p>
            </title>
            <aug>
               <au>
                  <snm>Jenssen</snm>
                  <fnm>TK</fnm>
               </au>
               <au>
                  <snm>Laegreid</snm>
                  <fnm>A</fnm>
               </au>
               <au>
                  <snm>Komorowski</snm>
                  <fnm>J</fnm>
               </au>
               <au>
                  <snm>Hovig</snm>
                  <fnm>E</fnm>
               </au>
            </aug>
            <source>Nat Genet</source>
            <pubdate>2001</pubdate>
            <volume>28</volume>
            <fpage>21</fpage>
            <lpage>28</lpage>
            <xrefbib>
               <pubidlist>
                  <pubid idtype="pmpid" link="fulltext">11326270</pubid>
                  <pubid idtype="doi">10.1038/88213</pubid>
               </pubidlist>
            </xrefbib>
         </bibl>
         <bibl id="B12">
            <title>
               <p>Using text analysis to identify functionally coherent gene groups</p>
            </title>
            <aug>
               <au>
                  <snm>Raychaudhuri</snm>
                  <fnm>S</fnm>
               </au>
               <au>
                  <snm>Schutze</snm>
                  <fnm>H</fnm>
               </au>
               <au>
                  <snm>Altman</snm>
                  <fnm>RB</fnm>
               </au>
            </aug>
            <source>Genome Res</source>
            <pubdate>2002</pubdate>
            <volume>12</volume>
            <fpage>1582</fpage>
            <lpage>1590</lpage>
            <xrefbib>
               <pubidlist>
                  <pubid idtype="pmpid" link="fulltext">12368251</pubid>
                  <pubid idtype="doi">10.1101/gr.116402</pubid>
                  <pubid idtype="pmcid">187532</pubid>
               </pubidlist>
            </xrefbib>
         </bibl>
         <bibl id="B13">
            <title>
               <p>Genomic Comparison of Salmonella enterica Serovars and Salmonella bongori by Use of an S. enterica Serovar Typhimurium DNA Microarray</p>
            </title>
            <aug>
               <au>
                  <snm>Chan</snm>
                  <fnm>K</fnm>
               </au>
               <au>
                  <snm>Baker</snm>
                  <fnm>S</fnm>
               </au>
               <au>
                  <snm>Kim</snm>
                  <fnm>CC</fnm>
               </au>
               <au>
                  <snm>Detweiler</snm>
                  <fnm>CS</fnm>
               </au>
               <au>
                  <snm>Dougan</snm>
                  <fnm>G</fnm>
               </au>
               <au>
                  <snm>Falkow</snm>
                  <fnm>S</fnm>
               </au>
            </aug>
            <source>J Bacteriol</source>
            <pubdate>2003</pubdate>
            <volume>185</volume>
            <fpage>553</fpage>
            <lpage>563</lpage>
            <xrefbib>
               <pubidlist>
                  <pubid idtype="pmpid" link="fulltext">12511502</pubid>
                  <pubid idtype="doi">10.1128/JB.185.2.553-563.2003</pubid>
                  <pubid idtype="pmcid">145314</pubid>
               </pubidlist>
            </xrefbib>
         </bibl>
         <bibl id="B14">
            <title>
               <p>Complete genome sequence of Salmonella enterica serovar Typhimurium LT2</p>
            </title>
            <aug>
               <au>
                  <snm>McClelland</snm>
                  <fnm>M</fnm>
               </au>
               <au>
                  <snm>Sanderson</snm>
                  <fnm>KE</fnm>
               </au>
               <au>
                  <snm>Spieth</snm>
                  <fnm>J</fnm>
               </au>
               <au>
                  <snm>Clifton</snm>
                  <fnm>SW</fnm>
               </au>
               <au>
                  <snm>Latreille</snm>
                  <fnm>P</fnm>
               </au>
               <au>
                  <snm>Courtney</snm>
                  <fnm>L</fnm>
               </au>
               <au>
                  <snm>Porwollik</snm>
                  <fnm>S</fnm>
               </au>
               <au>
                  <snm>Ali</snm>
                  <fnm>J</fnm>
               </au>
               <au>
                  <snm>Dante</snm>
                  <fnm>M</fnm>
               </au>
               <au>
                  <snm>Du</snm>
                  <fnm>F</fnm>
               </au>
               <au>
                  <snm>Hou</snm>
                  <fnm>S</fnm>
               </au>
               <au>
                  <snm>Layman</snm>
                  <fnm>D</fnm>
               </au>
               <au>
                  <snm>Leonard</snm>
                  <fnm>S</fnm>
               </au>
               <au>
                  <snm>Nguyen</snm>
                  <fnm>C</fnm>
               </au>
               <au>
                  <snm>Scott</snm>
                  <fnm>K</fnm>
               </au>
               <au>
                  <snm>Holmes</snm>
                  <fnm>A</fnm>
               </au>
               <au>
                  <snm>Grewal</snm>
                  <fnm>N</fnm>
               </au>
               <au>
                  <snm>Mulvaney</snm>
                  <fnm>E</fnm>
               </au>
               <au>
                  <snm>Ryan</snm>
                  <fnm>E</fnm>
               </au>
               <au>
                  <snm>Sun</snm>
                  <fnm>H</fnm>
               </au>
               <au>
                  <snm>Florea</snm>
                  <fnm>L</fnm>
               </au>
               <au>
                  <snm>Miller</snm>
                  <fnm>W</fnm>
               </au>
               <au>
                  <snm>Stoneking</snm>
                  <fnm>T</fnm>
               </au>
               <au>
                  <snm>Nhan</snm>
                  <fnm>M</fnm>
               </au>
               <au>
                  <snm>Waterston</snm>
                  <fnm>R</fnm>
               </au>
               <au>
                  <snm>Wilson</snm>
                  <fnm>RK</fnm>
               </au>
            </aug>
            <source>Nature</source>
            <pubdate>2001</pubdate>
            <volume>413</volume>
            <fpage>852</fpage>
            <lpage>856</lpage>
            <xrefbib>
               <pubidlist>
                  <pubid idtype="pmpid" link="fulltext">11677609</pubid>
                  <pubid idtype="doi">10.1038/35101614</pubid>
               </pubidlist>
            </xrefbib>
         </bibl>
         <bibl id="B15">
            <title>
               <p>Genes encoding putative effector proteins of the type III secretion system of Salmonella pathogenicity island 2 are required for bacterial virulence and proliferation in macrophages</p>
            </title>
            <aug>
               <au>
                  <snm>Hensel</snm>
                  <fnm>M</fnm>
               </au>
               <au>
                  <snm>Shea</snm>
                  <fnm>JE</fnm>
               </au>
               <au>
                  <snm>Waterman</snm>
                  <fnm>SR</fnm>
               </au>
               <au>
                  <snm>Mundy</snm>
                  <fnm>R</fnm>
               </au>
               <au>
                  <snm>Nikolaus</snm>
                  <fnm>T</fnm>
               </au>
               <au>
                  <snm>Banks</snm>
                  <fnm>G</fnm>
               </au>
               <au>
                  <snm>Vazquez-Torres</snm>
                  <fnm>A</fnm>
               </au>
               <au>
                  <snm>Gleeson</snm>
                  <fnm>C</fnm>
               </au>
               <au>
                  <snm>Fang</snm>
                  <fnm>FC</fnm>
               </au>
               <au>
                  <snm>Holden</snm>
                  <fnm>DW</fnm>
               </au>
            </aug>
            <source>Mol Microbiol</source>
            <pubdate>1998</pubdate>
            <volume>30</volume>
            <fpage>163</fpage>
            <lpage>174</lpage>
            <xrefbib>
               <pubidlist>
                  <pubid idtype="pmpid" link="fulltext">9786193</pubid>
                  <pubid idtype="doi">10.1046/j.1365-2958.1998.01047.x</pubid>
               </pubidlist>
            </xrefbib>
         </bibl>
         <bibl id="B16">
            <title>
               <p>Host-pathogen interactions: Host resistance factor Nramp1 up-regulates the expression of Salmonella pathogenicity island-2 virulence genes</p>
            </title>
            <aug>
               <au>
                  <snm>Zaharik</snm>
                  <fnm>ML</fnm>
               </au>
               <au>
                  <snm>Vallance</snm>
                  <fnm>BA</fnm>
               </au>
               <au>
                  <snm>Puente</snm>
                  <fnm>JL</fnm>
               </au>
               <au>
                  <snm>Gros</snm>
                  <fnm>P</fnm>
               </au>
               <au>
                  <snm>Finlay</snm>
                  <fnm>BB</fnm>
               </au>
            </aug>
            <source>Proc Natl Acad Sci U S A</source>
            <pubdate>2002</pubdate>
            <volume>99</volume>
            <fpage>15705</fpage>
            <lpage>15710</lpage>
            <xrefbib>
               <pubidlist>
                  <pubid idtype="pmpid" link="fulltext">12441401</pubid>
                  <pubid idtype="doi">10.1073/pnas.252415599</pubid>
                  <pubid idtype="pmcid">137780</pubid>
               </pubidlist>
            </xrefbib>
         </bibl>
      </refgrp>
   </bm>
</art>
