siRecords is a continuing effort aimed to document all mammalian siRNA experiments reported in literature, and provide systematically rated efficacies for these experiments 33. Currently, about 9000 records of siRNA experiments targeting more than 3000 genes are hosted in the siRecords database. For each siRNA experiment, we document the siRNA sequence, the target gene, key information about experimental conditions (cell line used; the method of producing the siRNA – chemically synthesized or vector-based; the method of testing the siRNA efficacy – western blot or real-time PCR or others), and an efficacy rating (elaborated below).

For this investigation, we picked all complete records of 19-mer siRNA experiments (21-mers if the two overhanging nucleotides on the 3' ends are counted) from the siRecords collection (dated 12/12/2005). The distribution of number of records per study is highly skewed – about 17.5% of the records (657 siRNA experiments) originated from 0.4% of the studies (6 studies, each reporting ≥ 30 siRNA experiments, Figure 1). To prevent our analyses from being biased by this small number of studies, we limited the number of siRNA experiments originated from a single study to be ≤ 30. For these studies where more than 30 siRNA experiments were reported, we randomly picked 30 to include in our analyses and discarded the rest. The resulting dataset includes the records of 3277 siRNA experiments targeting 1518 genes originated from 1417 independent studies. We randomly divided the dataset into two subsets at a 2:1 ratio. The larger subset – termed Set A – included 2184 records, and was used to survey features significantly associated with high efficacies and analyze the combinatorial effects of these features. The other subset (termed Set T, 1093 records) was reserved to test the conclusions obtained through the analyses of Set A.

We set out to determine, using the Set A data, what "features" of the siRNA experiments are associated with elevated RNAi efficacies. A feature is a binary property of a siRNA experiment concerning a factor potentially relevant to siRNA efficacy, for example, the 6th nucleotide of the siRNA sequence = A. Each feature has a "complementary feature". A feature and its complementary feature constitute a "feature pair". More discussions about the definition of feature and related terms can be found in

Methods

In siRecords, the effectiveness of any siRNA experiment is rated on a four-level scale: very high (if the gene product was reduced by ≥ 90%), high (if the gene product was reduced by 70–90%), medium (if 50–70% knock-down was achieved); and low (if < 50% knock-down was observed). In Set A, the percentages of records receiving very high, high, medium and low efficacy ratings are 34.1%, 34.6%, 16.3% and 14.9% respectively (Figure 2). The decision of using this four-level rating scheme was made based on balanced considerations about the usefulness and the reliability of the ratings 33. One consequence of this decision is that that the conventional t-test type of analysis 11 can not be performed on this dataset, because the dependent variable (efficacy rating) is not a continuous variable, but rather a categorical, ordinal variable. Proper categorical analysis techniques need to be adopted to analyze this type of data 35.

We chose to use the Wald test of monotone trend to assess the evidence that the presence of a feature is associated with a significant up-shift (or down-shift) of the efficacy distribution. In addition, we conducted odds ratio permutation tests for two efficacy levels: > 90% and > 70% efficacies, because in siRNA design practice, we are interested in assessing whether a feature leads to increased chances of achieving higher efficacies (see

Methods

We examined 276 features (they constitute 138 "feature pairs") for their association with higher RNAi efficacies, using the Wald test of monotone trend and the odds ratio permutation tests. The features we examined include, to our knowledge, all that have been implicated in previous studies to improve siRNA effectiveness. Each of these features can be placed into one of five categories. The first category is based on nucleotide identities at specific positions on the 19-mer siRNA sequence, e.g. the 6th nucleotide = A; there are 76 feature pairs in this category. The second category includes 19 feature pairs that are either composite sequence features, e.g. there are at least three (A/U)'s in the seven nucleotides at the 3' end of the siRNA, or features that are defined based on the G/C content of the siRNA. The third category consists of 13 feature pairs that are based on the thermodynamics of the siRNAs as measured by the melting temperature, or binding energy. The fourth category, consisting of 16 feature pairs, includes features based on target mRNA sites, such as the relative positions of the target sites on the mRNA, and the local secondary structures of the target regions. Finally, the fifth category includes 14 feature pairs that are based on experimental settings, such as the cell lines used in the experiments (HeLa cells, HEK293 cells, and others), the methods used for making and delivering the siRNAs, and the methods used to evaluate the efficacy of the siRNA (Western blot, PCR-based, and others). The complete list of these tested features, and references to the studies that implicated them in enhancing siRNA efficacies, are provided in Supplementary Tables 1-5 in Additional file 1.

Of the features examined, we found 34 that were associated with a significant improvement in the efficacy distribution (P < 0.01, Wald test of monotone trend; FDR controlled at 0.056 by the q-value technique 36); among which, 26 significantly elevated the chance of achieving > 90% efficacies (P < 0.01, odds ratio permutation test, FDR controlled at 0.038), and 27 significantly enhanced the probability of achieving > 70% efficacies (P < 0.01, odds ratio permutation tests, FDR controlled at 0.044; see Supplementary Tables 1-5 in Additional file 1). There are several cases of sub-feature – super-feature relationships among these significant features. For example, the features the 6th nucleotide = A, and the 6th nucleotide ≠ C were both significant features, however, the former is a sub-feature of the latter since when the former feature is present, the latter must also be present. In each occurrence of sub-feature – super-feature relationship, we eliminated all but the one feature determined to be the most significant by the Wald test. The feature the 6th nucleotide = A was thus eliminated because the Wald test P value of this feature was higher than that of the feature the 6th nucleotide ≠ C. G/C content related features were treated as a special case. Several different G/C content ranges were suggested in previous studies as being possibly associated with high RNAi effectiveness (32–79%, 30–70%, 30–52%, 35–60%, 20–50% and 31.6–57.9%) 11121824303132. All these features were tested. Although they do not constitute sub-feature – super-feature relationships, we treated these features as redundant features, and retained only one of them (G/C content is between 35 and 60%) because it yielded the lowest P value (0.00018) in the Wald test. The resulting list of non-redundant significant features is shown in Table 1. Detailed discussions about these significant features, and comparisons of our analyses with previous findings can be found in the Additional file 1.

The presence of any single significant feature was not sufficient to improve the efficacy distribution substantially. When present alone, the significant features listed in Table 1 increased the probability of achieving > 90% efficacies by an average of only 2.5% (from 34.1% to 36.6%), and they increased the chance of achieving > 70% efficacies by an average of merely 2.2% (from 68.7% to 70.9%). To achieve substantially improved efficacies, the concurrent presence of several significant features is required.

When multiple features are co-present, we cannot assume that their contributions to the effectiveness of the RNAi experiments are additive, since features are not always independent of one another. For instance, the presence of the feature the 19th nucleotide = (A/U), clearly increases the probability that the feature there are at least three (A/U)'s in the seven nucleotides on the 3' end of the siRNA to be true. Indeed, these two features exhibited negative cooperativity: when present alone, they increased the chances of achieving > 90% efficacies by 2.6% and 2.4%, respectively; when co-present, these two features resulted in merely a 2.7% increase in the chance of achieving > 90% efficacy, much smaller than the sum of the effects of the two features (see Additional file 1 for discussions about cooperativity and additive effects of multiple features).

In seeking effective siRNA design rules, we should try to identify combinations of features that exhibit positive cooperativity. The large size and diverse origins of the records in the siRecords dataset allowed us to systematically analyze how features jointly influence siRNA efficacies. Three significant features: Cell line = HeLa, Test method = Western blot and Test object ≠ mRNA were excluded from joint effect analyses because they are based on experimental settings, which are typically chosen independent of siRNA design. For the remaining 17 significant features, we looked at all possible combinations of a fixed number (l = 2,3,4,5 and 6) of features. For each combination of l features, we examined the number of records in Set A that concurrently carry all l features, and the percentages of these records that achieved > 90% and > 70% efficacies. For every given l, we focused on the top-10 feature combinations, i.e., the 10 combinations that exhibited the highest percentage of records achieving > 90% or > 70% efficacies. When there was a tie of more than 10 feature combinations, all tied combinations were considered. As we expected, as l – the number of features in the combinations increased, the number of records concurrently carrying all l features declined sharply (Figure 3C). Meanwhile, the percentage of experiments achieving > 90% and > 70% efficacies increased steadily as l, the number of features included in the feature combinations, increased (Figure 3A and 3B).

The sigmoid shape of the two ascending curves is an indication of positive cooperativity (see discussion in Additional file 1). This suggests that by simply retaining the feature combinations that led to the highest percentages of records achieving efficacies of > 90% or > 70%, we were, in effect, exploiting the positive cooperativity, or favorable interaction, among these features. At l = 5, 24 feature combinations had a 100% chance of having efficacies > 70%, that is, every experiment in which the siRNA used had all features contained in any one of the 24 feature combinations exhibited efficacies of > 70%. Similarly, 14 feature combinations had 100% probabilities of having efficacies > 90% at l = 5, meaning that all siRNA experiments having these feature combinations demonstrated efficacies > 90%. At l = 6, 188 feature combinations had 100% probabilities of having efficacies of > 70%, and 94 feature combinations had 100% probabilities of achieving efficacies of > 90%.

A disjunction of the top feature combinations described above (across l = 2 through 6; a feature combination is also called a rule thereafter) defines a rule set for designing effective siRNA experiments. Rule sets defined in this way are likely to contain redundancies, because if a rule consisting of l^ MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGSbaBgaqcaaaa@2E1D@ features {f₁, f₂,..., fl^ MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqWGMbGzdaWgaaWcbaGafmiBaWMbaKaaaeqaaaaa@2F9E@} is one of the best l^ MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGSbaBgaqcaaaa@2E1D@ -feature combinations, then a rule consisting of (l^ MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGSbaBgaqcaaaa@2E1D@+1) features {f₁, f₂,..., fl^ MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqWGMbGzdaWgaaWcbaGafmiBaWMbaKaaaeqaaaaa@2F9E@, f₀}, where f₀ is any other feature, is likely to be one of the best (l^ MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGSbaBgaqcaaaa@2E1D@+1)-feature combinations thus is also selected into the rule set. A disjunctive rule merging (DRM) algorithm can be applied to remove redundancies of the rule sets, in the mean time allowing the control over the stringency of the resulting rule sets (see

Methods

We assessed the performance of the DRM rule sets, and compared it with that of 15 existing online design tools commonly used in siRNA design practice, using the Set T data reserved for this purpose (Table 3 and Figure 4). Set T includes the records of 1,093 siRNA experiments, representing 1,014 unique target sites on 744 genes. How do we assess the performance of a siRNA design program? A good siRNA design program should (a) provide a sufficient number of candidate siRNAs for a given gene; and (b) offer a high PPV (positive predictive value), or a low false positive rate (see

Methods

On the number of candidate siRNAs predicted, the DRM rule set with the highest stringency level (RS_0.951) produced on average 18.9 predicted effective siRNAs per gene. This indicates that this rule set offers sufficient candidate siRNAs in an ordinary siRNA design task for a gene of an average length. However, the smallest number of predicted effective siRNAs for a gene is 1. This suggests that for genes of the shortest lengths, the number of candidate siRNAs offered by this rule set may not be enough. There are considerations other than achieving high efficacy (e.g., avoiding cross-reactivity with other genes) in the design of siRNA experiments, thus it is desirable to have multiple candidate siRNAs designed for every gene. For genes of the shortest lengths, we resort to DRM rule sets of lower stringency levels. For example, RS_0.845 produced at least 3 potentially effective siRNAs for each gene, and an average of 38.1 potentially effective siRNAs per gene (see Supplementary Figure 3 in Additional file 1). The online design tools varied greatly in the numbers of candidate siRNAs they provided. The highest number of predicted effective siRNAs was offered by EMBOSS sirna by Institute Pasteur (639.4 siRNAs per gene). IDT RNAi Design by IDT, Inc. produced the lowest number of predicted effective siRNAs (5.8 siRNAs per gene). Among the 15 online design tools, 10 offered larger numbers of candidate siRNAs than DRM RS_0.951, and 4 provided larger numbers of candidate siRNAs than DRM RS_0.845.

Given that a sufficient number of candidate siRNAs are provided, the most important parameter that measures the performance of a design tool is the PPV. Only a small proportion of possible siRNA sites have been experimentally tested for effectiveness (1,014 sites among 2,453,510 possible 19-mer sites on the 744 genes). Based on these experimentally tested siRNA sites, we compared the PPVs of the DRM rule sets to those of 15 existing online design tools. For the "> 90% efficacy" setting and "> 70% efficacy" setting, DRM RS_0.951 showed PPVs of 72.7% and 90.9%, respectively. In other words, 72.7% of the predicted effective siRNAs by DRM RS_0.951 had > 90% efficacy, and 90.9% of the predicted effective siRNAs showed > 70% efficacy. This rule set and two others with lower α level, RS_0.895 and RS_0.845 surpassed all online design tools in PPVs on both settings. Among the 15 online design tools, the three that offered the highest PPVs for the "> 90% efficacy" setting were WI siRNA Selection Program by Whitehead Institute (53.8%), siDESIGN Center by Dharmacon Inc. (48.5%) and BLOCK-iT RNAi Designer by Invitrogen Corp. (47.6%), respectively; and the four that offered the highest PPVs for the "> 70% efficacy" setting were siRNA Target Finder by GenScript Corp. (85.2%), WI siRNA Selection Program by Whitehead Institute (84.6%), siDESIGN Center by Dharmacon. Inc. (81.8%) and siRNA Target Designer by Promega Corp. (81.8%), respectively.

Set T is a fair dataset to be used for the purpose of performance comparison between the DRM rule sets and the online design tools, because it contains no overlapping records with Set A, based on which the DRM rule sets were derived. However, Set T might not be considered as a completely independent dataset, because (a), there are records in Set T that originated from the same studies as some records in Set A; and (b), there are records of siRNA experiments in Set T that target the same genes as some experiments in Set A. To rule out the possibility that these two factors might contribute to better performance of the DRM rule sets for unforeseen reasons and unfairly favor the DRM rule sets in the performance comparison, we compiled an "independent subset" of Set T, eliminating all records that share the same origins of any records in Set A, and all records that target the same genes that are also targeted by any records in Set A. We compared the performance of the DRM rule sets with that of the 15 online design tools using this independent subset (including 224 siRNAs targeting 197 different genes, see Table 4). Because of the reduced size of the dataset (by nearly 80%), the sensitivity, specificity and PPVs for all tools and rule sets showed higher levels of variability. The three DRM rule sets with the highest α levels: RS_0.951, RS_0.895 and RS_0.845 achieved 100% PPV. Two online design tools, BLOCK-iT by Invitrogen Corp. and WI siRNA Selection Program by Whitehead Institute also achieved 100% PPV, but the other online design tools achieved lower PPVs that range between 50.0% and 86.4%. Although the small size of the independent subset prevented this analysis from being completely conclusive, it is fair to state that the comparison made based on the independent subset is generally in agreement with the comparison made based on the entire Set T.

All records of 19-mer siRNAs (not counting the overhanging nucleotides on the 3' end) were retrieved from the siRecords database. The records that failed to meet the following criteria were excluded from further analyses: (1) had complete annotations of cell line types, test methods, transfection methods and efficacy classification; (2) had target mRNA lengths ≤ 16,000 nucleotides (this is a limit set by the Mfold program for calculation of thermodynamics features, see below); (3) the siRNA sequence had no mismatches with the targeted site by pair-wise Blast (NCBI bl2seq v.2.2.9, parameters "-p blastn -W 7 -q -1 -F F"). For studies where more than 30 siRNA experiments were reported, we randomly chose 30 to include in our analyses. The cell line types and test methods were grouped based on ATCC (American Type Culture Collection) 43 and Protocol Online 44, respectively.

We define a feature as a binary property of a siRNA experiment concerning a factor potentially influencing the efficacy of the experiment. For a given siRNA experiment, any defined feature is either present or absent. Some example features are listed below:

(1) The 6th nucleotide of the siRNA sequence (counting from the 5' end on the sense strand) is an adenine (A).

(2) The 17th nucleotide of the siRNA sequence is not a guanine (G).

(3) There are at least three (A/U)'s in the seven nucleotides on the 3' end.

(4) The G/C content of the siRNA sequence is between 30 and 52%.

For Features (1) and (2), the concerning factors potentially influencing the siRNA efficacy are the identities of the 6th and the 17th nucleotides of the siRNA sequence, respectively. For Feature (3), the concerning factor is the seven nucleotides as a whole on the 3' end of siRNA sequence. For Feature (4), the concerning factor is the G/C content of the siRNA sequence.

Each feature has a complementary feature, that is, the alternative property concerning the same factor. For instance, the complementary feature of Feature (1) is the 6th nucleotide of the siRNA sequence ≠ A; and the complementary feature of Feature (3) is there are at most 2 (A/U)'s in the seven nucleotides on the 3' end. For any given siRNA experiment and any given feature, either the feature holds true for the experiment, or the complementary feature must hold true. A feature and its complementary feature constitute a feature pair.

For a given factor, there are multiple ways of formulating features. In some cases, the so-called sub-feature – super-feature relationships can result. For example, the following four features are all concerned with same factor – the identity of the 6th nucleotide of the siRNA sequence:

(5) The 6th nucleotide = A.

(6) The 6th nucleotide ≠ A.

(7) The 6th nucleotide =C.

(8) The 6th nucleotide ≠ C.

Wherever Feature (5) is present, Feature (8) must also be present. Thus, Feature (5) is a sub-feature of Feature (8), and Feature (8) is a super-feature of Feature (5). Similarly, Features (7) and (6) also constitute a pair of sub-feature – super-feature relationship.

We surveyed 276 features (constituting 138 feature pairs) in this study. These features can be classified into the following five categories:

Determined by the four-level scheme used to rate the efficacy of siRNA experiments, proper categorical analysis techniques were needed to analyze these data. For any given feature, we calculated the efficacy distribution (among the four levels – very high, high, medium and low) of all siRNA experiments carrying this feature, and compared it with the efficacy distribution of all siRNA experiments carrying the complementary feature of this feature. Chi-square (χ²) test of independence is a commonly used test that finds evidence of difference between two discrete distributions. However, this test assumes that the dependent variable (efficacy rating) is a nominal variable rather than an ordinal variable, thus it is not able to tell us whether the presence of a feature results in higher or lower efficacy. A more appropriate test will find evidence of monotone trend, that is, whether the presence of a feature is associated with a significant up-shift or down-shift of the efficacy distributions among the four levels. Consider the joint probability distribution {π_{i, j}} between the presence/absence of a particular feature (which defines i: i = 1 if the feature is present, and i = 0 if the feature is absent), and the four-level efficacy ratings (which defines j: j = 3 if efficacy rating = "very high", j = 2 if efficacy rating = "high", j = 1 if efficacy rating = "medium", and j = 0 if efficacy rating = "low"). We calculate the probabilities of concordance and discordance:

Π c = 2 ∑ i ∑ j π i j ( ∑ h > i ∑ k > j π h k ) , Π d = 2 ∑ i ∑ j π i j ( ∑ h > i ∑ k < j π h k ) . MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakqaabeqaaiabfc6aqnaaBaaaleaacqWGJbWyaeqaaOGaeyypa0JaeGOmaiZaaabuaeaadaaeqbqaaGGaciab=b8aWnaaBaaaleaacqWGPbqAcqWGQbGAaeqaaaqaaiabdQgaQbqab0GaeyyeIuoaaSqaaiabdMgaPbqab0GaeyyeIuoakiabcIcaOmaaqafabaWaaabuaeaacqWFapaCdaWgaaWcbaGaemiAaGMaem4AaSgabeaaaeaacqWGRbWAcqGH+aGpcqWGQbGAaeqaniabggHiLdGccqGGPaqkcqGGSaalaSqaaiabdIgaOjabg6da+iabdMgaPbqab0GaeyyeIuoaaOqaaiabfc6aqnaaBaaaleaacqWGKbazaeqaaOGaeyypa0JaeGOmaiZaaabuaeaadaaeqbqaaiab=b8aWnaaBaaaleaacqWGPbqAcqWGQbGAaeqaaaqaaiabdQgaQbqab0GaeyyeIuoaaSqaaiabdMgaPbqab0GaeyyeIuoakiabcIcaOmaaqafabaWaaabuaeaacqWFapaCdaWgaaWcbaGaemiAaGMaem4AaSgabeaaaeaacqWGRbWAcqGH8aapcqWGQbGAaeqaniabggHiLdGccqGGPaqkcqGGUaGlaSqaaiabdIgaOjabg6da+iabdMgaPbqab0GaeyyeIuoaaaaa@73D9@

Then we calculate the γ difference between these two probabilities:

γ = Π c − Π d Π c + Π d . MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaaiiGacqWFZoWzcqGH9aqpdaWcaaqaaiabfc6aqnaaBaaaleaacqWGJbWyaeqaaOGaeyOeI0IaeuiOda1aaSbaaSqaaiabdsgaKbqabaaakeaacqqHGoaudaWgaaWcbaGaem4yamgabeaakiabgUcaRiabfc6aqnaaBaaaleaacqWGKbazaeqaaaaakiabc6caUaaa@3E33@

The sample γ has approximately a normal distribution, with standard error calculated using the Delta method

σ 2 = 16 ( Π c + Π d ) 4 ∑ i ∑ j π i j [ Π d π i j ( c ) − Π c π i j ( d ) ] 2 , MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaaiiGacqWFdpWCdaahaaWcbeqaaiabikdaYaaakiabg2da9maalaaabaGaeGymaeJaeGOnaydabaGaeiikaGIaeuiOda1aaSbaaSqaaiabdogaJbqabaGccqGHRaWkcqqHGoaudaWgaaWcbaGaemizaqgabeaakiabcMcaPmaaCaaaleqabaGaeGinaqdaaaaakmaaqafabaWaaabuaeaacqWFapaCdaWgaaWcbaGaemyAaKMaemOAaOgabeaaaeaacqWGQbGAaeqaniabggHiLdaaleaacqWGPbqAaeqaniabggHiLdGccqGGBbWwcqqHGoaudaWgaaWcbaGaemizaqgabeaakiab=b8aWnaaDaaaleaacqWGPbqAcqWGQbGAaeaacqGGOaakcqWGJbWycqGGPaqkaaGccqGHsislcqqHGoaudaWgaaWcbaGaem4yamgabeaakiab=b8aWnaaDaaaleaacqWGPbqAcqWGQbGAaeaacqGGOaakcqWGKbazcqGGPaqkaaGccqGGDbqxdaahaaWcbeqaaiabikdaYaaakiabcYcaSaaa@62D1@

where

π i j ( c ) = ∑ i > h ∑ j > k π h k + ∑ h > i ∑ k > j π h k , π i j ( d ) = ∑ i > h ∑ j < k π h k + ∑ h > i ∑ k < j π h k . MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakqaabeqaaGGaciab=b8aWnaaDaaaleaacqWGPbqAcqWGQbGAaeaacqGGOaakcqWGJbWycqGGPaqkaaGccqGH9aqpdaaeqbqaamaaqafabaGae8hWda3aaSbaaSqaaiabdIgaOjabdUgaRbqabaaabaGaemOAaOMaeyOpa4Jaem4AaSgabeqdcqGHris5aaWcbaGaemyAaKMaeyOpa4JaemiAaGgabeqdcqGHris5aOGaey4kaSYaaabuaeaadaaeqbqaaiab=b8aWnaaBaaaleaacqWGObaAcqWGRbWAaeqaaaqaaiabdUgaRjabg6da+iabdQgaQbqab0GaeyyeIuoaaSqaaiabdIgaOjabg6da+iabdMgaPbqab0GaeyyeIuoakiabcYcaSaqaaiab=b8aWnaaDaaaleaacqWGPbqAcqWGQbGAaeaacqGGOaakcqWGKbazcqGGPaqkaaGccqGH9aqpdaaeqbqaamaaqafabaGae8hWda3aaSbaaSqaaiabdIgaOjabdUgaRbqabaaabaGaemOAaOMaeyipaWJaem4AaSgabeqdcqGHris5aaWcbaGaemyAaKMaeyOpa4JaemiAaGgabeqdcqGHris5aOGaey4kaSYaaabuaeaadaaeqbqaaiab=b8aWnaaBaaaleaacqWGObaAcqWGRbWAaeqaaaqaaiabdUgaRjabgYda8iabdQgaQbqab0GaeyyeIuoaaSqaaiabdIgaOjabg6da+iabdMgaPbqab0GaeyyeIuoakiabc6caUaaaaa@8321@

Let

z 2 = γ 2 σ 2 , MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqWG6bGEdaahaaWcbeqaaiabikdaYaaakiabg2da9maalaaabaacciGae83SdC2aaWbaaSqabeaacqaIYaGmaaaakeaacqWFdpWCdaahaaWcbeqaaiabikdaYaaaaaGccqGGSaalaaa@3706@

then z² is a Wald statistics that has a chi-squared null distribution with 1 degree of freedom, based on which a Wald test can be conducted to find significant monotone trend 35.

The monotone trend test finds evidence about whether the presence of a particular feature is associated with significant up-shift of the four-level efficacy distribution. If the evidence of such association is found, however, this test alone is not able to tell us where the up-shift takes place. In RNAi experiments, we are most concerned with the chances of achieving higher efficacies. Thus, we also conducted permutation tests of odds ratios for achieving > 90% and > 70% efficacies. In the siRecords data, the chance of achieving > 90% (or > 70%) efficacies can be approximated by the proportion of records bearing "very high" (or "high"/"very high") efficacy ratings. For a given feature, the odds ratio for > 90% efficacies, θ₉₀, is defined as

θ 90 = π 1 , > 90 / ( 1 − π 1 , > 90 ) π 0 , > 90 / ( 1 − π 0 , > 90 ) , MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaaiiGacqWF4oqCdaWgaaWcbaGaeGyoaKJaeGimaadabeaakiabg2da9maalaaabaGae8hWda3aaSbaaSqaaiabigdaXiabcYcaSiabg6da+iabiMda5iabicdaWaqabaGccqGGVaWlcqGGOaakcqaIXaqmcqGHsislcqWFapaCdaWgaaWcbaGaeGymaeJaeiilaWIaeyOpa4JaeGyoaKJaeGimaadabeaakiabcMcaPaqaaiab=b8aWnaaBaaaleaacqaIWaamcqGGSaalcqGH+aGpcqaI5aqocqaIWaamaeqaaOGaei4la8IaeiikaGIaeGymaeJaeyOeI0Iae8hWda3aaSbaaSqaaiabicdaWiabcYcaSiabg6da+iabiMda5iabicdaWaqabaGccqGGPaqkaaGaeiilaWcaaa@5639@

where π_{1, > 90} is the proportion of records bearing "very high" efficacy rating (i.e., with > 90% efficacies) in the subset of the experiments carrying the feature, and π_{0, > 90}is the proportion of records bearing "very high" efficacy ratings in the subset of the experiments carrying the complementary feature of the feature concerned. To generate a null distribution of the odds ratio, Set A was randomly split into two subsets, one of which was arbitrarily marked with "feature present", the other marked with "complementary feature present", and an odds ratio was calculated accordingly. This process was repeated 100000 times, and the 100000 resampled odds ratios constituted the null distribution. Given any feature to be tested, the P value was calculated as

P₉₀ = (i|θ90i MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqaH4oqCdaqhaaWcbaGaeGyoaKJaeGimaadabaGaemyAaKgaaaaa@31D8@ > θ₉₀)/100000,

where θ90i MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqaH4oqCdaqhaaWcbaGaeGyoaKJaeGimaadabaGaemyAaKgaaaaa@31D8@ is the ith resampled odds ratio, and θ₉₀ is the true odds ratio of the feature. The odds ratio permutation test for > 70% efficacies was conducted similarly, with the proportion of records bearing "very high" or "high" efficacy ratings substituted for that of records bearing "very high" ratings in the above description.

Meaningful statistics tests require the use of sufficiently large datasets. All features were subject to a "dataset size filter" using an arbitrarily set threshold of 30 records: if a given feature was carried by fewer than 30 records in Set A, then this feature and the complementary feature of this feature were excluded from the statistics tests and following analyses. Four features – GC stretches of length ≥ 9, G/C content is not between 30 and 79%, Cell line = T24 and Test method = Flow cytometry, as well as their complementary features were excluded for this reason.

The simultaneous testing of the large number of hypotheses requires the curbing of the type I error rate with the consideration of the "multiple testing" problem. We chose to control the FDR by taking the q-value approach 36, because of its ability to adapt to the true distribution of the input p-values. We used the "bootstrap" method, rather than the default "smoother" method (which is equivalent to Benjamini and Hochberg's FDR controlling method 62) in estimating the FDR, because U-shape distributions were observed for the input p-values for both the Wald test and the odds ratio permutation tests, likely introduced by the fact that one-sided tests were conducted when two-sided signals were present 63.

We define a rule as a conjunction of (l) features. An l-feature rule is also called an l-feature combination. A rule set is defined as a disjunction of (m) rules. Generally speaking, the larger m is, the higher sensitivity the rule set achieves, in the mean time, the lower specificity the rule set has to offer.

The disjunctive rule merging (DRM) algorithm was developed to remove the redundancy in the rule sets resulting from the combined effect analysis of multiple features, in the mean time exerting control over the stringency of the rule sets. The listing of the DRM algorithm is as follows.

Input: Θ: a set of disjunctive rules that contains redundancy; each rule, r_i, is a conjunction of m_i features: r_i = {f_i, f₂,..., fmi MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqWGMbGzdaWgaaWcbaGaemyBa02aaSbaaWqaaiabdMgaPbqabaaaleqaaaaa@3123@}, and is labeled with P_i = the proportion of records reaching > 90% efficacy in the subset of Set A records satisfying r_i.

α: stringency factor, with a range of [0, 1].

Initialization: Create rule set RS = φ.

Step 1:For every r_i ∈ Θ satisfying P_i ≥ α, add r_i into RS.

Step 2: For j = 2,3,...,5

      For any rule r_p ∈ RS where m_p = j

         For any rule r_q ∈ RS where m_q > j

            if r_p ⊂ r_q, then remove r_q from RS.

         End For

      End For

   End For

Output: RS: non-redundant set of disjunctive rules with stringency >α.

It is easy to see that given any α, the rule set resulting from the DRM algorithm (thus called a DRM rule set) is fixed. The reverse, however, is not true. A DRM rule set does not correspond to a single α value, but rather, a range of different α's. For example, the DRM rule sets for any α between 0.901 and 1 are exactly the same (containing 7 rules). We note this rule set as RS_0.951, where 0.951 is the mid-point of the range of α for which the rule sets are produced.

Naturally, the higher α level, the higher specificity the DRM rule set possesses; meanwhile, the lower sensitivity the rule set has to offer. Therefore, the DRM algorithm with variable α values allows us to choose the proper combination of sensitivity and specificity that suits our needs. In the siRNA design of a typical setting, we are most concerned with achieving high specificity, and can often tolerate lower sensitivity, since there is a large pool of possible target sites to choose from – for a mRNA of length w, in theory there are (w-19+1) target sites to pick from. Therefore, we are most concerned with the behavior of the rule sets with high (close to 1) α values.

Design tasks were performed for the 744 genes in Set T using the following 15 online siRNA design tools with the default settings.

Ambion siRNA Target Finder (Ambion, Inc.) 64. We used the mRNA sequence as the input. By default, no restriction of the ending dinucleotides was specified, and no restriction of the G/C content was specified. Occurrences of 4 or more identical nucleotides in a row were allowed.

Jack Lin's siRNA Sequence Finder (Cold Spring Harbor Laboratory) 65. We used the full-length mRNA sequence as the input. The spacer length was set as to be 19.

siDESIGN Center (Dharmacon, Inc.) 66. We used the mRNA sequence as the input. No restriction of the leading sequences was specified. The target region was limited to the ORF (open reading frame), the G/C content range was set as 30–52%, and the patterns "GGG" and "CCC" were excluded. The BLAST filtering option was turned on by default.

siRNA Target Finder (GenScript Corp.) 67. We provided the GenBank accession of the mRNA as the input. The length of siRNA was set to be 19. By default, the G/C content range was set to be between 30% and 60%, and sequence selection region was restricted to the ORF.

Imgenex sirna Designer (Imgenex Corp.) 68. The target mRNA was specified using the GenBank accession. The siRNA length was set to be 19. The parameter "nucleotide target" was set to be 50 by default. The parameter "first nucleotide target for siRNA" was set as "AA". The G/C content range was set to be between 45% and 51%. Occurrences of 4 identical A's or T's in a row, or 3 identical (C/G)'s in a row were not allowed. By default, the BLAST search was not performed.

EMBOSS siRNA (Institute Pasteur) 69. We used the full-length mRNA sequence as the input. By default, no restriction of the leading or ending dinucleotides was specified. Occurrences of 4 identical nucleotides in a row were allowed.

IDT RNAi Design (SciTools) (Integrated DNA Technologies, Inc.) 70. The mRNA sequence was provided as the input, and the "21mer" option was selected. The "Unified RNAi Rule Set" was used in the design. The G/C content range was set to be between 30% and 70%. The asymmetrical end stability base pair length was set to be 5. The 5' antisense asymmetrical end stability weight was set to be 0.5, and the 3' overhang was set to be "TT" by default. The default setting was also used for all motifs preferences.

BLOCK-iT RNAi Designer (Invitrogen Corp.) 71. We provided the mRNA sequence as the input. By default, the search in the target region was limited to the ORF. The minimum/maximum allowed G/C contents were set to be 35% and 55%, respectively. The BLAST search option was turned on by default.

siSearch (Karolinska Institutet) 72. We provided the mRNA sequence as the input. By default, the G/C content range was set to be between 30% and 60%. The candidate sites with scores of 6 or above were obtained. The minimum energy difference between two ends of the siRNA was set to be 0. Occurrences of 4 (A/U)'s in a row were not allowed, and the siRNAs containing immunostimulatory motifs were removed. The repeat masking was turned on by default.

SiMAX (MWG-Biotech, Inc.) 73. We used the Genbank accession to specify the target. By default, occurrences of > 3 identical nucleotides in a row in the siRNA sequences, or U's at the 3' end were not allowed. The G/C content range was set to be between 30% and 53%. The search range was restricted to the region between the 100th nucleotide downstream of the start codon and the 100th nucleotide upstream of the end codon. By default, BLAST filtering or secondary structure analysis was not performed.

BIOPREDsi (Novartis Institutes for BioMedical Research) 74. We used the mRNA sequence as the input. The number of predicted siRNAs was set to be 10.

Promega siRNA Target Designer (Promega Corp.) 75. We used the mRNA sequence as the input. The RNAi system was set to be the "T7 RiboMAX Express RNAi system". By default, the target length was set to be 19, and the search region was set to be the whole input sequence.

QIAGEN siRNA Design Tool (QIAGEN, Inc.) 76. We specified the mRNA sequence as the input. The option "Start siRNA sequence with AA" was turned on by default. The BLAST search was not performed.

SDS/MPI (University of Hong Kong) 77. We used the full-length mRNA sequence as the input. The option "MPI Principles" was selected. The filtering of ineffective siRNAs based on secondary structures was not performed. By default, the G/C content range was set to be between 30% and 70%, and the search region was restricted to ≥ 100 nucleotides downstream of the CDS.

Whitehead WI siRNA Selection Program (Whitehead Institute for Biomedical Research) 78. We used the mRNA sequence as the input. By default, the sequence pattern "AAN19TT" was searched for. The G/C content range was set to be between 30% and 70%. Occurrences of 4 or more identical T's, A's or G's in a row were not allowed. Occurrences of 7 or more consecutive (G/C)'s in a row were also not allowed. By default, the checking with BLAST was not performed.

The performance of a siRNA design rule set, or an online siRNA design tool, can be assessed by several parameters. Two of the most often used ones are specificity and sensitivity, as illustrated in Table 5. Specificity is defined as N_D/(N_B+ N_D); and sensitivity is defined as N_A/(N_A+ N_C). An ROC (Receiver Operative Characteristic) curve can be used to visually depict the overall performance of a rule set. The ROC curve is the plot of sensitivity vs. (1-specificity). Another parameter is the positive predictive value (PPV), defined as N_A/(N_A+ N_B). The PPV is a very important parameter in siRNA design practice, because it describes out of the siRNAs predicted to be effective, how big proportion turn out to be truly effective. The value (1-PPV) is sometimes called the "false positive rate".