The primary functionalities of XSTREAM, as shown in Figure 1, can be divided into five high level stages: Pre-Processing, TR Detection, TR Characterization, Post-Processing, and Output. For a technical description of the algorithm, presented within the same organizational context, refer to the Appendix section.

For processing by XSTREAM, input sequences must be in FASTA format. Valid sequences are sent to the seed detection module. XSTREAM searches the input sequence for short exact substring repeats, or seeds, of two or three sizes, depending on the input length (see 29 for an excellent example of the use of seeds, or k-tuple probes in TR detection). Seed pairs are used to provide starting points and potential periods for TR detection. The use of seeds allows XSTREAM to rapidly identify putative TRs. For every adjacent pair of matching seeds, XSTREAM records both the sequence distance between them and the sequence index of the leftmost seed. Each distance is a potential TR period.

Following seed detection, XSTREAM attempts to extend each seed pair. Two sequence iterators move downstream from each seed in a parallel manner, returning characters for comparison. Running totals of character match and mismatch are kept. We define i as the amount of character matching required between two tandemly arranged words in order for them to be designated a TR. For example, if i is set to 0.8, then at least 80% of the aligned characters among two words at a given period must be identical. Seed extension always stops when for any seed pair, the iterator for the leftmost seed collides with the rightmost seed. If at any point during the procedure, the character mismatch count divided by the current potential period exceeds or equals 1 - i, seed extension is aborted, thereby reducing running time. Similarly, seed extension is prematurely terminated if the match count becomes sufficiently high. To include indels during seed extension, we use a novel heuristic, which is presented in the Appendix section.

Each candidate TR resulting from successful seed extension is subjected to further expansion using the same basic mechanism as seed extension. XSTREAM examines sequence space both downstream and upstream of the current candidate domain using increments equal to the TR period. Potential repeat copies are evaluated by comparing new sequence space with the reference repeat, which is the leftmost repeat resulting from the initial seed extension. If indels are allowed and if domain expansion using seed extension fails to agree with i, we invoke a second strategy. The second approach, termed GRDP (see Appendix), can more accurately perform a subsequence pairwise comparison at the expense of slightly increased running time. A novel feature of our implementation is the user's ability to limit the maximum width of the dynamic programming (DP) matrix (parameter g), resulting in θ(n) time and space complexities for global pairwise alignments.

Following domain expansion, we instantiate a procedure called maximality. Employing a user-adjustable scoring scheme, maximality finds the longest stretch of characters both downstream and upstream that can legitimately be added to each candidate TR. This procedure is invoked because TRs in nature do not always occur in integer copy numbers and XSTREAM's TR domain expansion method is limited to integer copies.

Finally, XSTREAM masks input sequence space corresponding to each maximally extended candidate TR. Sequence masking prevents further seed extensions in sequence regions that constitute TR domains, thus functioning to prevent output redundancy as well as reduce running time. For details of sequence masking, refer to Redundancy Elimination I as well as Two-stage TR detection in the Appendix.

To further refine each candidate TR, XSTREAM segments every TR domain into its component copies. Parsing can be accomplished by a trivial subdivision of the TR domain using the current period, an optimal subdivision using wrap-around dynamic programming (WDP, 31), or a heuristic subdivision using GRDP. For details about implementation and when each method is invoked, refer to the Appendix section.

Following TR parsing, each TR undergoes a multiple alignment of its copies. A procedure identical in concept to STAR Alignment is used when indels are allowed. Because practical running time is emphasized in our implementation, pairwise sequence comparisons during STAR Alignment may be computed in a non-optimal manner using GRDP.

Following multiple alignment of each TR, a consensus sequence is computed. Each consensus is democratically derived using the majority rule. In addition, XSTREAM computes an error associated with the consensus – the lower the error, the stronger the agreement between the consensus and its represented domain. We define I as the minimum allowable matching between the consensus and the aligned TR for the TR to be reported to the user. For example, if I equals 0.8, then the consensus error cannot exceed 0.2 or 20% disagreement.

Next, XSTREAM inspects the edges of each aligned TR domain (with TR copy number greater than 2) for accordance with the consensus. If either edge mismatches with the consensus, that edge is truncated. Since all TRs must have at least 2 copies, edge trimming is not performed on TR domains with TR copy number = 2.

Occasionally, because of matching considerations, TR domains are identified with periods that are reducible. Therefore, the last step of TR Characterization functions to reduce overestimated TR periods (see Redundancy Elimination II in the Appendix).

XSTREAM attempts to merge sufficiently similar TRs that either overlap in the input sequence or are in close enough proximity to one another. To compute sufficient similarity, XSTREAM invokes the concept of cyclical permutations, which enables effective consensus sequence comparison (see Merging and Consensus Comparison in the Appendix). As a result, XSTREAM can identify TR domains with large regions of indels and/or substitutions that, without merging, would be reported as separate TRs. This procedure is thus important for detecting rapidly evolving TR sequences.

Following merging, XSTREAM invokes a series of finalizing functions called finishing touches, which serve to fine-tune the characterization of each TR domain as well as remove TRs that are insufficiently fit for output. TR characterization refinement involves rerunning maximality, redoing multiple alignment, rerunning reducibility, and looking for nested TRs (see Appendix). After additional characterization, finishing touches removes TRs with unacceptable amounts of overlap (see Redundancy Elimination III in the Appendix). Finally, remaining TRs are tested for agreement with user-defined filtration criteria.

All TRs that satisfy the output criteria are sent to the consensus comparison (CC) module. CC clusters TRs on the basis of consensus similarity. By ordering TRs by consensus sequence homology in the output, XSTREAM reduces output redundancy while facilitating the identification of TR families from the input dataset. Related TRs may reflect structural or functional homology of their corresponding protein sequences. The current implementation of CC only compares TRs of equal period.

XSTREAM automatically generates HTML files in a format similar to the output from Tandem Repeats Finder (TRF) 29. HTML output 1 contains a TR summary table and list of TR information, including sequence positions, period, and copy number. The range of sequence positions for each TR is hyperlinked to HTML output 2, which displays TR multiple alignments and consensus sequences. In the case of a multiple sequence input, XSTREAM generates HTML output 3, which reports a list of all input sequences containing reported TRs. An additional output option is a colored TR schematic, in PNG or HTML format, that represents the modular architectures of TR-containing sequences. The main user-definable output parameters of XSTREAM are presented in Table 1. A list of all user-defined parameters can be found on the XSTREAM webserver 32.

• S = input sequence, which takes values from alphabet {A,C,G,T} for nucleotide sequences and alphabet {A,C,D,E,F,G,H,I,K,L,M,N,P,Q,R,S,T,V,W,Y} for proteins

· |S| = length of S

· S[j] = the character at index j in S with j ≥ 0

· S[i, j] = the subsequence in S from index i to index j inclusively

• Xi = TR domain i

· |Xi| = length of entire TR domain Xi

· Xi[j] = repeat copy j in Xi with j ≥ 0

· |Xi[j]| = length of copy j

· |Xi[]| = size of array Xi[]

· XiS = lowest index of Xi; starting position in S

· XiE = highest index of Xi; ending position in S

· XiSE = index range [XiS, XiE]

· Ei = copy number (exponent) of Xi

· Ci = consensus sequence of Xi

· Pi = period of Xi = period of Ci

· CEi = consensus error of Xi =

- Without gaps: # of mismatching characters to consensus/total # of characters in aligned Xi

- With gaps: see Consensus Building

· Ii = indel error of Xi = # of gaps in aligned Xi/total # of characters in aligned Xi

· Ri = referential repeat copy of Xi: used during TR domain expansion and maximality

• {X} = {X₀, X₁,..., X_n} = set of all identified TR domains

To find repeats of various periods in any FASTA-formatted input sequence S, XSTREAM looks, by default, for exact repetitions (seeds) of lengths 3 and 5. Length 7 is also used if |S| ≥ 2000. Seed lengths are user-adjustable. XSTREAM records the distance between each pair of adjacent seeds, |p - q|, where the lowest index in S of each seed in the pair is represented by p and q respectively, and p <q. All seed positions and distances between adjacent seeds are stored and accessed using a hash table. In addition, XSTREAM records in an integer array M, the hashcodes and sequence indices for all seeds of minimum length L, where L = 3 by default. For instance, a seed of length L starting in position 5 in S would have its hashcode stored in M[5]. The utility of M is explained shortly.

A major obstacle to efficient alignment of gapped TRs is dynamic programming (DP), which, for global pairwise-sequence alignment, has time complexity O(n²), where n = TR period. Because optimal alignment of TR copies may, in some cases, place a temporal burden on the user, we explored heuristic options. We decided to implement a non-optimal variant of pairwise global sequence alignment DP, which we call gap-restricted DP (GRDP). GRDP requires a user-modifiable parameter, g, which governs the maximum number of consecutive gaps that can be used during GRDP pairwise alignment. Because of g, the maximum traceable width of the DP matrix is held constant for all periods, is equal to 2g + 1, and is symmetrically distributed with respect to the main diagonal. As a result, GRDP has space complexity θ(n) and time complexity θ(n), enabling a 1:1 correspondence between increasing period and running time. The following recursion describes GRDP:

s c o r e ( i , j ) = max ⁡ { s c o r e ( i − 1 , j ) + g a p s c o r e ( i , j − 1 ) + θ s c o r e ( i + 1 , j − 1 ) + g a p MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqWGZbWCcqWGJbWycqWGVbWBcqWGYbGCcqWGLbqzcqGGOaakcqWGPbqAcqGGSaalcqWGQbGAcqGGPaqkcqGH9aqpcyGGTbqBcqGGHbqycqGG4baEdaGabaqaauaabaqadeaaaeaacqWGZbWCcqWGJbWycqWGVbWBcqWGYbGCcqWGLbqzcqGGOaakcqWGPbqAcqGHsislcqaIXaqmcqGGSaalcqWGQbGAcqGGPaqkcqGHRaWkcqWGNbWzcqWGHbqycqWGWbaCaeaacqWGZbWCcqWGJbWycqWGVbWBcqWGYbGCcqWGLbqzcqGGOaakcqWGPbqAcqGGSaalcqWGQbGAcqGHsislcqaIXaqmcqGGPaqkcqGHRaWkiiGacqWF4oqCaeaacqWGZbWCcqWGJbWycqWGVbWBcqWGYbGCcqWGLbqzcqGGOaakcqWGPbqAcqGHRaWkcqaIXaqmcqGGSaalcqWGQbGAcqGHsislcqaIXaqmcqGGPaqkcqGHRaWkcqWGNbWzcqWGHbqycqWGWbaCaaaacaGL7baaaaa@779E@

Note that depending on g, we place constraints on where each score possibility can be computed. The parameters gap and θ denote gap penalty and match/mismatch values respectively. By default, all DP procedures use values gap = -4, mismatch = -4, and match = 2. An example of GRDP alignment is shown in Figure 8. Also, note that if g = 0, XSTREAM completely disallows gaps, and thus the decision to allow insertions/deletions (indels) is left up to the user. By default, g = 3. In addition, our implementation of GRDP requires input sequences of equal length. In cases where input sequences have different lengths, both sequences are made the same length by appending gap characters to the shorter sequence. Any columns with two gaps in the resulting pairwise alignment are removed. Since standard DP is practical in many situations, several functions of XSTREAM toggle GRDP on and off depending on projections of time complexity. GRDP is used in four major functions: TR domain expansion, TR parsing, multiple alignment, and consensus comparison.

XSTREAM implements three strategies to eliminate two types of TR redundancy – reducible TR periods and TR domain overlap.

I) XSTREAM searches for TRs in order of increasing period. As TRs are found, their corresponding sequence space is flagged, preventing further searching in processed sequence regions. This tactic combats both kinds of redundancy and reduces running time.

II) To combat reducible TR periods, XSTREAM is rerun on the consensus sequence of each TR domain from the input sequence. (see Figure 1) If the consensus sequence Ci of Xi contains a TR domain xi that spans Ci's entire length, XSTREAM repartitions Xi using the consensus of xi, resulting in Xi*, whose period is an even multiple of Xi's period. Xi* is retained and Xi erased (Xi*→Xi) if Xi* passes the user-adjustable TR filtration criteria.

III) The following redundancy elimination method, invoked by default, functions to remove TR domain overlap. The user can control the execution and parameters of this method because it may not always be desirable to remove TR domain overlap and because we are convinced that the amount of reasonable overlap among TR domains is an arbitrary matter. We now state the rules that determine whether for a given TR pair Xi and Xj, XSTREAM deletes one or neither. The rules are enforced in the order they are presented; i.e. rule set (i) must fail to move to rule set (ii) and so on. Let I = |XiSE ∩ XjSE| (length of intersection of TR domains i, j). By default, α = .9, β = .75, γ = .9, and δ = .6.

i) if (α·|Xi|) ≤ I ≤ |Xi| and |Xi| ≤ |Xj|

   if |Xi| < (β·|Xj|)

      delete Xi

   else if |Xi| < (γ·|Xj|) and (Ei <Ej or CEi > CEj)

         delete Xi

      else if |Xi| ≥ (γ·|Xj|) and Ei <Ej

            delete Xi

         else delete Xj

ii) Same as (i) but swap i and j

iii) if I ≥ (δ·max(|Xi|, |Xj|))

   if CEi ≥ CEj and Ei ≤ Ej

      delete Xi

   else delete Xj

iv) if I ≥ (δ·min(|Xi|, |Xj|))

   delete min(|Xi|, |Xj|)

As shown in Table 1, XSTREAM allows the user to restrict the TR period range. If MinP <T and MaxP ≥ T, TR detection proceeds in two phases, where phase I examines periods = T, and phase II examines periods <T. By default, T = 10. This procedure reduces the frequency of inconsistent results. We now describe our reasoning.

As mentioned in Redundancy Elimination I, TRs are identified in order of increasing period and sequence space is masked for every successful seed extension. Because of these two facts and because the value of MinP can be altered, it is possible to differentially characterize the same TR domain Xi, or perhaps miss Xi altogether, for the case Pi ≥ max[all tested MinP values]. This problem can occur because as XSTREAM moves up the period ladder toward Pi, different stretches of sequence space may be removed in and around Xi for different values of Min. We determined empirically that by first scanning upward from a short period, such as 10, we could greatly mitigate this problem. To illustrate, see Figure 2 for an example of a TR domain containing many short period TRs. Without Two-Stage TR Detection, this period 152 TR domain would not be reported since most of its sequence space would be masked by its constituent TRs.

Following completion of phase I, all masked sequence space is reset to unused, thereby allowing shorter period TRs to be found independently of longer period TRs. Redundancy removal strategies II and III are invoked later and will remove any redundancy caused by XSTREAM's two-stage TR detection procedure.

To ensure pragmatic running time for all possible periods, we implemented a heuristic that governs seed extension for periods greater than or equal to 1000 characters. If |S| ≥ 2000, during seed detection, an additional hashcode array M* is kept, which stores hashcodes and sequence positions for seeds of maximum length L*, which by default is 7. Then, for every pair of seeds with distance ≥ 1000, XSTREAM initially invokes a filtration step, which jumps across M* a user-defined number of times t and looks for matching hashcodes. This method is identical to seed extension as described earlier, except that S is not used and x is incremented by floor(d/t) after each hashcode comparison. Thus, if g > 0, CW can be invoked. XSTREAM runs standard seed extension and TR domain expansion (using M* and L*) on periods ≥ 1000 if and only if t* matches are recorded during the filtering phase, where t* = t/3. Therefore, seed pairs with distances ≥ 1000 are subjected to a quick and preliminary filter, which although imperfect, drastically reduces running time for input sequences on the chromosome size scale. By default, t = 20.

Within each TR consensus sequence, XSTREAM searches for nested TRs – TRs that occur within TRs. This is a novel feature in the domain of protein analysis software and may provide important information about primary sequence architectures and peptide TR evolution. For a given Xi, we define a nested TR as a TR present in Ci that does not span Ci's entire length. Since TR degeneracy can complicate identifying nested structures, XSTREAM only looks for nested TRs in consensus sequences. Our procedure detects nested TRs of unlimited nesting depth, with no gaps and no mismatches. This algorithm employs a top-down approach to locating TRs, as opposed to the bottom-up method used by XSTREAM. A top-down approach is useful for nested TRs because it identifies the longest period TR first, then in a recursive manner, restarts the algorithm within that TR, and continually digs deeper until no more TRs can be found. By working off the greedy assumption that the longest period TRs are the best candidates for nesting, we avoid issues of TR overlap inherent in the bottom-up strategy. The main drawback to our nesting method is its time complexity, which is O(n³), where n = Pi. We therefore restrict this method to TRs from {X} with periods ≤ 1000 and only find nested TRs with periods ≤ 300. We set the minimum nested TR period at 1 for proteins and 2 for nucleotide sequences. The time complexity is O(n³) due to the worst-case scenario of comparing subsequences of all possible sizes in all possible sequence regions.

XSTREAM implements a user-adjustable divide and conquer procedure to reduce memory consumption. If enabled, the input sequence is segmented into overlapping fragments of length l prior to TR detection. The last fragment of the input sequence may be of length <l. Overlapping regions have length l*, which is equivalent to the maximum detectable TR period. After all fragments are processed, the set of identified TRs are directed to the merging procedure, which functions to both extend TRs across fragment boundaries and consolidate overlapping regions. By default, l = 100,000 and l* = 10,000.