The following description is focused on the essential features and algorithms of HitKeeper. The installation and a number of technical details are explained in-depth in the Reference Manual, which is part of the distribution package.

HitKeeper is a collection of scripts that interact as concurrent clients with a relational database management system (RDBMS). The software is mostly written in Perl and was developed and tested under various flavours of Linux and Mac OS X using MySQL as RDBMS; it is currently being extended to other RDBMS.

Fig. 1 is a schematic representation of the logical organization of HitKeeper. Starting from the abstract concept of "data", the structure is built around three kinds whose properties are reflected in the organization of the software and that are hardcoded in the application: seq, biological sequences; mot, motifs for predicting hits on the sequences, and cla, hierarchical classification (currently limited to taxonomy).

For each kind, HitKeeper allows multiple types of data to be dealt with. As an example, seq allows multiple types of sequence data, such as "pep" (for peptide) and "nuc"(for nucleotide). Similar to this, mot may comprise the type "pattern" as well as "profile" or "HMM". All types, and all computation algorithms between them (e.g. which program is used to run a pattern search on a peptide sequence), are defined in a central configuration file. Besides some general parameters (database server, etc.), this file also holds the list of the modules and external programs that are used (a) for parsing the flat files, (b) performing the actual computations, and (c) dispatching and/or mirroring to any external computing elements. Custom modules written in Perl can be added, either derived from the existing modules or written "from scratch". Parsing of the input data is based on lazy parsing that extracts only the relevant information. This minimizes the amount of maintenance that might be induced by format changes in the source data. Custom modules can also be provided for the mirroring of the databases to external computing elements (e.g. formating for a BLAST server).

Five distinct clients are available. Three of them, HKLoader, HKUpdater and HKPublisher, are used for RDBMS housekeeping and control of the data flow. They operate concurrently in the background, similar to a system daemon. The two other scripts, HKReader and HKAdmin, are used to interact with the RDBMS. While the former is solely intended for querying the system, the latter also allows the administration of HitKeeper; as an example, the HitKeeper administrator defines interactively which database (UniProt, Prosite, etc.) is actually parsed and used for the calculations. Both scripts provide the functionality of a command-line tool, and the interactivity of a "shell-like" environment; they accept input from STDIN and can thus be controlled through other scripts and pipes. This allows automation and enables performing tasks in batch mode, either directly from the command line or by reading commands from a file.

As mentioned above, HitKeeper reads three kinds of input data. Each is associated with a "pipeline" where several versions of a database, such as weekly releases, can coexist. However, only the version with the status 'current' is in the production stage and can be queried. Fig. 2 illustrates how the seq and mot pipelines are synchronized with respect to the incremental updates of the hit list.

Computations are set up on a per-database basis, so that all entries in a given sequence database are expected to be calculated against all entries in a motif database. However, not all databases are necessarily calculated against each other: the software uses a "subscription" model, defining which database pairs are to be calculated. In this way, it is possible to set up calculations as needed and to adjust the allocation of computing resources.

All hit list computations are performed by calling external software, i.e. they are not hardcoded in HitKeeper. A simple implementation of a pattern-matching algorithm is provided and can be used as template for custom extensions.

If a sequence or motif database is updated, repeating the same computations for sequences or motifs that have not changed should be avoided. This is the purpose of the incremental update algorithm in HitKeeper. The algorithm is identical for sequence and motif database updates. Complications arise from the optional subscriptions and from the handling of redundancy; a typical case handled by our algorithm is outlined in Fig. 3.

The prerequisites for the installation of HitKeeper are the availability of a MySQL server and a few Perl modules from CPAN; according to our experience, the presence of the system administrator is preferable at this stage. The deployment of a HitKeeper installation as such is essentially performed through a shell script within a few seconds.

The validation of a HitKeeper installation concerns in particular the incremental updates and the query mechanism. Two tests are provided in the distribution and implemented as shell scripts, thus emulating commands as they would be typed by the user instead of querying the RDBMS directly. They should be run as "operational qualification tests" and will verify the correct behaviour of the parser, computation engine, incremental update, and query mechanism.

Historically, HitKeeper was developed as the "back end" of the MyHits web site 2 as depicted in Fig. 4. MyHits has been in production status since 2003 and currently handles more than 7 million non-redundant sequences with weekly updates from a number of major databases, and 21 million hits on these (Table 1). Thus, HitKeeper can be considered to be robust and scalable.

HitKeeper implements an elaborate and original query syntax to retrieve information. Besides support for logical operators (OR, AND, NOT), HitKeeper allows sequences to be retrieved with logical constraints on the arrangement of the motifs found along the sequence. An expression that specifies such a particular "motif of motifs" is called a metamotif. Metamotif queries are expressed in a grammar that is specific to HitKeeper, yet human readable. This grammar is parsed and then compiled into an SQL query. The metamotif query language was inspired by our experience with mmsearch 3 and twofeat 4.

While presenting the full query capabilities of HitKeeper is out of the scope of this paper, some typical examples of the query language are given below. The setup of the following example dataset is described fully in the Reference Manual; hits can be calculated overnight on a standard Linux workstation. We make use of three common databases: The Swiss-Prot protein sequences (designated with sw hereafter) 5, the Prosite patterns (pat) 6, and the NCBI taxonomy data (taxid) 78. An additional database of "virtual" motifs is automatically derived from the Swiss-Prot "FT" lines (feature) with a script that produces more than 800 of these 'virtual motifs'. The computation yields two 'hit' lists (Swiss-Prot vs Prosite, and Swiss-Prot vs the ft motifs), and a single 'hat' list (i.e. match between sequence and classification): Swiss-Prot vs NCBI taxomomy. The latter is used for filtering by taxonomy.

The original text entries can be retrieved using alternative but unique designations. As an example, the Prosite entry for the pattern with id CORNICHON can be retrieved using its name, ID, or accession number:

   mot_fetch_entry pat:CORNICHON

   mot_fetch_entry PS01340

Queries can be "stored" using a query identifier, indicated by -ref=... in the examples below. These identifiers are used to repeat, refine or even string together queries. The following example will refer to all bird sequences from Swiss-Prot:

   hat_query cla_parent=Aves seq_source=sw -ref=$BIRDSEQ

Re-using the same query, count the sequences and the species, then retrieve the non-redundant sequences of all birds in Swiss-Prot:

   query_stat $BIRDSEQ

   seq_fetch_nr $BIRDSEQ

Since the taxonomy data are available, it is easy to list all species covered by that query:

   cla_fetch_desc $BIRDSEQ

A list of all matches of Prosite patterns against these sequences can be obtained as follows:

   hit_query seq_name=$BIRDSEQ mot_source=pat

HitKeeper also has the capability to perform negative matching, such as finding all bird sequences with no match by any Prosite pattern:

   hit_query seq_source=sw mot_source=pat -ref=$PROSITE

   seq_query seq_name=$BIRDSEQ  not_seq_name=$PROSITE

A simple example is to retrieve all existing hits for the protein VAV_RAT [Swiss-Prot:P54100]:

   hit_query seq_name=sw:VAV_RAT

Note that these include DOMAIN sh2 (one hit), DOMAIN sh3 (two hits) in a particular arrangement. To search all sequences that fulfill a similar arrangement, a metamotif query with the followed by operator ~~ is used:

   mom_query (DOMAIN_sh3~~ DOMAIN_sh2~~ DOMAIN_sh3)

At the time of writing, there are about 22 proteins meeting this criterion in Swiss-Prot.

As another example, Prosite has a pattern pat:THIOREDOXIN that targets the active site of the thioredoxin domain [Prosite:PS00194]. In Swiss-Prot, the thioredoxin domain itself is annotated and was extracted from the FT line as ft:DOMAIN_Thioredoxin. The hit by the pattern is usually present within the domain annotation, but not always. In addition, some domain annotations do not include an active site that the pattern would match. The analysis is not straightforward since many proteins have multiple hits with the pattern and domain annotations. The following commands were used to obtain the counts as shown in Table 2:

First, a hit query for hits by either of the two motifs pat:THIOREDOXIN or ft:DOMAIN_Thioredoxin is performed and is saved under the query identifier $all_hit. The comma between the two motif names is the OR operator:

   hit_query -ref=$all_hit  mot_name=pat:THIOREDOXIN,ft:DOMAIN_Thioredoxin

Next a search for hits is carried out where the motif pat:THIOREDOXIN is 'embedded' in the motif ft:DOMAIN_Thioredoxin. Since there are proteins with multiple thioredoxin domains, a metamotif with the is included operator /</ was used to associate the pattern and the annotation. As this is a metamotif query, mom query is used instead of hit query:

   mom_query (pat:THIOREDOXIN) /</ (ft:DOMAIN_Thioredoxin)   -ref=$mom_hit

In a third step, hits that contain the motif pat:THIOREDOXIN but that are not included in those that bind the metamotif are identified:

   hit_query mot_name=pat:THIOREDOXIN  not_hit_list=$mom_hit -ref=$pat_not_mom

and the last dataset consists of hits with the motif ft:DOMAIN_Thioredoxin, but that are not in the metamotif hit list:

   hit_query mot_name=ft:DOMAIN_Thioredoxin   not_hit_list=$mom_hit -ref=$ft_not_mom

Finally, the results of all four queries are reported:

   query_stat $all_hit $mom_hit $ft_not_mom  $pat_not_mom

The result is summarised in Table 2 and shows that there are 81 matches by patterns that are not included in the matches by domains. On the other hand, there are 72 domains where the corresponding pattern is not present. The execution time of this last example is typically only a few seconds.