Several sources of noise perturb the acquisition step: signal over noise ratio in the images depends on the position of the colony within the imaging field (boundary effect), colonies can be hard to segment on the pictures, fluorophore emission spectra partially overlap as emissions "leak" into adjacent channels. Moreover synthesis efficiency is limited and therefore, within each colony, some DNA strands incorporate a non-complementary base or are de-synchronized because they failed to incorporate a nucleotide at a previous step. Both effects lead to the emission of a different fluorophore than the majority of the colony. These effects are possibly dependent on the base composition of the sequence17 and are obviously deteriorating with each additional chemistry cycle.

We use the sequencing of the phiX174 (see Material and Methods) to analyze the signal in the four color channels as the sequencing progresses. We first observe that the distribution of intensities in the individual channels shows a good separation between background noise and signal, although the shape of the histograms strongly depends on the dye used (Fig. 1A and Additional file 1). For example, G has a tighter dynamical range than T and the range generally decreases with the cycle number. The largest range spans 4–5 logs. As the sequencing progresses, dynamic range decreases, signal over noise ratios worsen and the separation between background noise and signal becomes increasingly blurred (Additional file 1). Next, we observe that the A and C channels, as well as the T and G channels, are highly correlated (Fig. 1A).

As observed above, there are three main sources of systematic bias at the level of intensity data. The first is the cross-talk between color channels: for example the A and C channels are not independent. Thus we transformed the raw intensities by a linear mapping to the basis with axes at angles ϕ and θ with respect to the original axes (cf. methods). We optimize the two angles so as to minimize the overall correlation between the transformed coordinates. We repeat this operation at each cycle of sequencing as well as with the other two, G and T channels (Fig. 1B).

The second important bias is the colony dephasing: the amount of fluorescence emitted in a particular channel at cycle n depends on the number of corresponding bases present in the sequence at positions 1, ..., n-1 because incorporation failures accumulated from previous cycles will be partly compensated at cycle n thereby increasing the signal in all channels. This cross-cycle dependence can be modelled by a binomial distribution with parameter q which is the probability of not elongating the complementary strand at each cycle of synthesis. We assume that this rate is equal for all nucleotides and all cycles. We determine the value of q by minimizing the average correlation between intensities at cycle n and n+1.

The last major source of systematic variation is due to an optical effect: on each tile, the colonies near the center of the image appear brighter than the ones near the edges (Additional file 2). We correct this by fitting a two-dimensional lowess 18 model to the intensities for each tile and subtracting the difference between the fit and the median intensity.

The three corrections are applied sequentially (cf. Methods) to the raw intensities before applying the model-based clustering algorithm described next.

We used a model-based clustering algorithm1219202122 to classify the intensity quadruples into four groups. Clearly, four well-delineated clusters corresponding to the four bases emerge (Fig. 1A–B). Specifically, we model the intensities measured in each channel by a mixture of four 4-dimensional Gaussian random variables and we use the intensity quadruples from all colonies in one or few combined tiles to fit the model parameters. The fitted model provides four probability distributions on the space of intensity quadruples, namely the probability P_A(k) = P(A|I₁(k), ..., I₄(k)) that the k^th base to call is an A knowing the measured intensities in all four channels at cycle k, and similarly for P_C, P_G and P_T. We can measure the level of uncertainty in our base calling by the entropy h(k)=−∑α∈{ACGT}Pα(k)log2Pα(k) MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaemiAaGMaeiikaGIaem4AaSMaeiykaKIaeyypa0JaeyOeI0YaaabuaeaacqWGqbaudaWgaaWcbaGaeqySdegabeaakiabcIcaOiabdUgaRjabcMcaPiabcYgaSjabc+gaVjabcEgaNnaaBaaaleaacqaIYaGmaeqaaOGaemiuaa1aaSbaaSqaaiabeg7aHbqabaGccqGGOaakcqWGRbWAcqGGPaqkaSqaaiabeg7aHjabgIGiolabcUha7jabbgeabjabboeadjabbEeahjabbsfaujabc2ha9bqab0GaeyyeIuoaaaa@5034@ which measures the uncertainty (in bits) in the determination of the correct k^th base23. Knowing h and the four probabilities we then use cutoffs in the probability simplex to decide which IUPAC code to call (Figure 2A, Methods). As the sequencing progresses, we also compute the cumulative entropy of each colony, H(n)=∑k=1,…,nh(k) MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaemisaGKaeiikaGIaemOBa4MaeiykaKIaeyypa0ZaaabuaeaacqWGObaAcqGGOaakcqWGRbWAcqGGPaqkaSqaaiabdUgaRjabg2da9iabigdaXiabcYcaSiablAciljabcYcaSiabd6gaUbqab0GaeyyeIuoaaaa@3F34@, which estimates the log₂ of the number of actual sequences compatible with the codes called up to position n. This total entropy is used to rank tags from least to most ambiguous. Figure 3A shows that this ambiguity score correlates with, but is markedly different from the Solexa fast-q quality score. The ambiguity metric is useful for genome assembly or polymorphism identification by allowing down-weighting the low quality tags when deriving statistics from multiple alignments of tags. As shown below, this metric can also be used to optimize tag lengths and increase the chance of identifying a match on the reference genome.

To assess the quality of our base calling and to compare it with the sequences obtained via Solexa's analysis pipeline, we compute the mapping efficiency #{reads mapping exactly to the genome}/#{total number of reads}. We used the fetchGWI tool 24 to search for unique exact matches of each sequenced tag encoded in the IUPAC code on the 5386 nt reference phiX174 genome sequence [RefSeq:NC_001422]. We thus discard every tag that matches at more than one position or does not match exactly anywhere on the reference sequence. One lane (330 tiles) of the Solexa flow cell produced 8 M tags, 3 M unique tags and 3.8 mappable tags, which amounts to a throughput of 137 million immediately usable bases per run. Sorting tags by decreasing quality we see (Figure 4) that low-entropy tags are easily identified by both the Solexa and Rolexa pipelines, but that the coverage achieved by Rolexa-called tags increases significantly among the low-quality sequences and results in an increased total coverage of up to 10–25% (average 15%). We also see that ranking by quality (or entropy, data not shown) is a judicious prioritization strategy since the coverage increase is sharp in the top part of the list and subsequently plateaus off.

To estimate error rates of sequencing, we used align0 25 to search for an optimal match between each tag and the phiX genome, and then computed the number of mismatches between tag and reference. Figure 5A shows how the error rates increases as a function of the sequencing cycles for Solexa tags. Rolexa tags called with the most probable ACGT base showed a slower increase, and introducing IUPAC codes significantly decreased both the intercept and slope of the error rate as a function of the sequencing cycle.

A surprising property of Solexa sequences is the imbalance between complementary A and T base counts as well as between G and C14. As shown in Figure 5B, there is progressive deterioration in the proportions as the sequencing progresses, which is likely related to the varying noise levels across fluorescent dyes for complementary base pairs as well as dye-specific chemical effects (see Fig. 1). In consequence an intensity close to the background is more likely to be called T than A, or C than G. Applying our corrections at the level of intensities stabilizes the proportions of bases, which is particularly pronounced for the T's. For reasons we do not currently understand the A/T ratio is not exactly one but stabilizes around 0.9 (Figure 5B).

To ascertain whether our increased coverage is not simply the consequence of the more degenerate alphabet, we verified that introducing ambiguities at random positions does not similarly improve the mapping. We thus selected the tags that did not match on the genome based on Solexa base calling, but did match after Rolexa introduced one to five ambiguous bases. Then we introduced ambiguities in these tags, with the same frequency as Rolexa, but at random positions. Figure 5D shows that only about 2% of those randomized mutations found a match on the genome, indicating that the entropy is a specific predictor of ambiguous positions.

While Solexa's quality score tends to decrease along the sequence, its distribution mostly spreads, rather than shifts, downwards (Fig. 3B). Computing a global length cutoff based on the average quality will therefore discard a lot of high-quality bases and not necessarily ensure a uniform quality. Thus we expect to increase the number of tags that can be mapped to a reference sequence by cutting them to a shorter length 26. However this procedure has a downside since it will reduce the coverage length per tag and increase the probability of finding multiple matches. Similarly, standard Solexa procedures suggest selecting tags with high average fast-q. Yet, a low average can be the result of just a few uncertain bases near the end of an otherwise useful tag.

We tested the different selections by applying the following quality filters. For the Solexa method we cut the tags at length 20, 25, 26, 28, 30, and then filtered all sequences with average fast-q score bellow 30, 25, or 20. In comparison, we used the following filtering procedure for Rolexa tags: we chose 3 different length-dependent entropy cutoffs IT(k) (see methods) and searched within each read for the longest k-mer with total entropy less than IT(k). We then extended this subsequence in both directions up to the next ambiguous base and eventually removed all tags shorter than 10 bases. The coverage statistics for the different filters are summarized in Figure 6. We performed a similar analysis of the 330 tiles of the sequencing of targeted human genomic regions and found an average of 50% increase in nucleotide coverage (Additional file 3). We see that the efficiency of Rolexa is superior in all datasets as measured by the ratio of actual coverage to expected coverage as well as by the ratio of tags having a unique match on the genome. The latter criterion is important since in many application of high-throughput sequencing (such as gene expression measures or ChIP-Seq), the extent of the coverage is less important than the number of hits on the genome. Similarly, in genotyping and targeted re-sequencing, where inexact matches are expected, the ability to reliably filter out low-quality tags before doing the matching to the reference sequence is of the highest importance, since actual polymorphisms must be distinguished from sequencing errors.

The phiX174 Control Library used was prepared by Illumina (Cat. No CT-901-1001). Briefly, the double-stranded covalently closed circular form of the viral DNA was broken into 100–400 bp fragments by nebulization; the ends repaired with Klenow, T4 DNA polymerase and PNK; and a base A was added on the 3'ends. After ligation of the double-stranded genomic adapters the sample was gel-purified to isolate fragments with "inserts" of approximately 200 bp and amplified by 18 cycles of PCR (Illumina protocol "Preparing Samples for Sequencing Genomic DNA", Part # 11251892 Rev. A). The library is quality controlled by cloning an aliquot into a TOPO plasmid and capillary sequencing 5–10 clones.

DNA Colonies were prepared by using a "Standard Cluster Generation Kit" (Cat. No. FC-103-1001) and 35 cycles of isothermal amplification in the flow-cell on the "Illumina Cluster Station" using a pM dilution of the 10 nM library. After amplification, one of the strands is removed; the free 3'-ends are blocked by terminal transferase in presence of dideoxynucleotides; and the genomic sequencing primer hybridized. The flow-cell was transferred to the Genome Analyzer "classic" and sequencing was performed for 36 cycles using a "36 Cycle Sequencing Kit" (Cat. No FC-104-1003) with the version 2.0 of the scanning buffer.

The samples used for Additional file 3 came from the pooled DNA obtained by long-range PCR amplification30 of a 30 kb region of chromosome 19 from 3 different individuals plus a 50 kb region of chromosome 3 from a fourth individual. Sequencing was performed as described above for the phiX174.

All data analysis for this paper has been performed with the R statistical framework http://www.r-project.org/ and the Rolexa package. This package uses the mclust routines20 as well as the fork package to run efficiently on multi-core architectures. Matching of short tags onto the genome have been performed with the fetchGWI tool24 by first generating a comprehensive index of the phiX174 genome and matching each query with its index entry. We used align0 25 to search for best matches from tags to the genome and estimate error rates (see Fig. 5A). When counting errors, an alignment of IUPAC code with one of its compatible bases was counted as correct match.

Raw data analysis (image analysis, initial base calling and fast-q scores) used the Firecrest image analysis module and the Bustard base-caller from the Illumina software suite (SolexaPipeline-0.2.2.6). No filtering or analysis with Gerald was performed.

We model the measured intensities I(α, n, x) (α = A, C, G, T is the dye channel, n = 1, ..., 36 is the cycle number and x denotes the colony coordinates) as the following combination of unbiased intensities J(α, n, x):

I ( α , n , x ) = ∑ m = 1 , ... , n ∑ β = A , C , G , T M ( α , β ) J ( β , m , x ) R ( m , n ) , MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaemysaKKaeiikaGIaeqySdeMaeiilaWIaemOBa4MaeiilaWIaemiEaGNaeiykaKIaeyypa0Zaaabuaeaadaaeqbqaaiabd2eanjabcIcaOiabeg7aHjabcYcaSiabek7aIjabcMcaPiabdQeakjabcIcaOiabek7aIjabcYcaSiabd2gaTjabcYcaSiabdIha4jabcMcaPiabdkfasjabcIcaOiabd2gaTjabcYcaSiabd6gaUjabcMcaPaWcbaGaeqOSdiMaeyypa0JaemyqaeKaeiilaWIaem4qamKaeiilaWIaem4raCKaeiilaWIaemivaqfabeqdcqGHris5aaWcbaGaemyBa0Maeyypa0JaeGymaeJaeiilaWIaeiOla4IaeiOla4IaeiOla4IaeiilaWIaemOBa4gabeqdcqGHris5aOGaeiilaWcaaa@64BE@

where the 4 × 4 matrix M is a mixture matrix which is block diagonal and depends on the 4 parameters ϕ_AC, θ_AC, ϕ_GT and θ_GT:

M ( { A , C } , { A , C } ) = ( c o s θ A C sin ⁡ θ A C cos ⁡ φ A C sin ⁡ φ A C ) , MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaemyta00aaeWaaeaadaGadaqaaiabdgeabjabcYcaSiabdoeadbGaay5Eaiaaw2haaiabcYcaSmaacmaabaGaemyqaeKaeiilaWIaem4qameacaGL7bGaayzFaaaacaGLOaGaayzkaaGaeyypa0ZaaeWaaeaafaqabeGacaaabaGaei4yamMaei4Ba8Maei4CamNaeqiUde3aaSbaaSqaaiabdgeabjabdoeadbqabaaakeaacyGGZbWCcqGGPbqAcqGGUbGBcqaH4oqCdaWgaaWcbaGaemyqaeKaem4qameabeaaaOqaaiGbcogaJjabc+gaVjabcohaZjabeA8aQnaaBaaaleaacqWGbbqqcqWGdbWqaeqaaaGcbaGagi4CamNaeiyAaKMaeiOBa4MaeqOXdO2aaSbaaSqaaiabdgeabjabdoeadbqabaaaaaGccaGLOaGaayzkaaGaeiilaWcaaa@5E4C@

and similarly for the G, T block, and the dephasing matrix R is a function of the parameter q and has a binomial structure:

R ( m , n ) = { 0  if  m > n , ( n m ) q n − m ( 1 − q ) m  if  m ≤ n . MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaemOuaiLaeiikaGIaemyBa0MaeiilaWIaemOBa4MaeiykaKIaeyypa0ZaaiqaaeaafaqabeGabaaabaGaeGimaaJaeeiiaaIaeeyAaKMaeeOzayMaeeiiaaIaemyBa0MaeyOpa4JaemOBa4MaeiilaWcabaWaaeWaaeaafaqabeGabaaabaGaemOBa4gabaGaemyBa0gaaaGaayjkaiaawMcaaiabdghaXnaaCaaaleqabaGaemOBa4MaeyOeI0IaemyBa0gaaOGaeiikaGIaeGymaeJaeyOeI0IaemyCaeNaeiykaKYaaWbaaSqabeaacqWGTbqBaaGccqqGGaaicqqGPbqAcqqGMbGzcqqGGaaicqWGTbqBcqGHKjYOcqWGUbGBcqGGUaGlaaaacaGL7baaaaa@5893@

The parameters ϕ_AC, θ_AC, ϕ_GT, θ_GT are determined by minimizing the following function:

F_n(θ_AC, ϕ_AC, θ_GT, ϕ_GT) = cor(M^-1I (A, n, •), M^-1 I(C, n, •))² + cor(M^-1 I(G, n, •), M^-1I(T, n, •))²,

which defines an intermediate intensity matrix K = M^-1 I. This is then introduced into the function

G ( q ) = ∑ α , n c o r ( R − 1 K ( α , n , • ) , R − 1 K ( α , n + 1 , • ) ) 2 , MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaem4raCKaeiikaGIaemyCaeNaeiykaKIaeyypa0ZaaabuaeaacqGGJbWycqGGVbWBcqGGYbGCdaqadaqaaiabdkfasnaaCaaaleqabaGaeyOeI0IaeGymaedaaOGaem4saSKaeiikaGIaeqySdeMaeiilaWIaemOBa4MaeiilaWIaeyOiGCRaeiykaKIaeiilaWIaemOuai1aaWbaaSqabeaacqGHsislcqaIXaqmaaGccqWGlbWscqGGOaakcqaHXoqycqGGSaalcqWGUbGBcqGHRaWkcqaIXaqmcqGGSaalcqGHIaYTcqGGPaqkaiaawIcacaGLPaaadaahaaWcbeqaaiabikdaYaaaaeaacqaHXoqycqGGSaalcqWGUbGBaeqaniabggHiLdGccqGGSaalaaa@5A73@

which is minimized to determine q.

Lastly, we correct systematic bias in function of the cluster coordinate as follows: we fit a 2-dimensional lowess 18 as a function of (x, y) coordinates and then subtract the difference between that fit and the median intensity across all four channels, for each tile and cycle.

We used the EEV model of the mclust algorithm20 to fit the Gaussian mixtures used to assign base probabilities in function of the four-dimensional intensity vector, similar as what was performed in 12. This model assumes Gaussian mixtures with four covariance matrices of the same shape and volume but with varying orientation. We initialize the classification by attributing each colony to the nucleotide with the highest (corrected) intensity. Given that initial classification, an M step of the mclust algorithm is performed which estimates the maximum likelihood parameters given the class attributions, where the parameters to estimate are the global scale and shape parameters as well as the centers and orientations of each class (using the covariance parameterization described in 20). This is then followed by an E step of the EM algorithm to estimate the conditional probabilities of each data point belonging to each class given the parameters estimates obtained previously. Full convergence of the EM algorithm is offered as an option but occasionally runs into spurious optima due to the effect of outliers (similarly to what was observed in 12). Further details of the implementation can be found in the package documentation (see Availability section).

The Rolexa algorithms require two types of cutoffs, which can both be easily user-defined in the Rolexa package. In the analyses presented, the limits between the different IUPAC bases in the probability simplex (Figure 2A) were set to HT(n) = log₂(n+0.5) with n = 1,2,3 (Figure 2B). Secondly the length-dependent cutoffs IT(n) were used to filter out uncertain bases by selecting the longest sub-tag S with total entropy smaller than IT(n = length (S)). In Figure 6 we used the following 6 choices: constants IT_c(n) = c with the constant c set to 2, 4, 6, or 8, and two cutoffs increasing with the tag length: IT_Log (n) = log₂ (4 + (n - 1)/5) and IT_Exp (n) = 2^(1+(n-1)/36). The latter two cutoffs interpolate between 2 and approximately 4 over the length of the sequence, but the first cutoff is concave (increases faster at the beginning) and the second is convex.

We have developed an R package called Rolexa which is freely available from http://bbcf.epfl.ch/Software. It is distributed under the GPL license and uses the mclust package which is part of the R distribution.