571 PRB, 2220 Pierce Avenue, Department of Biostatistics, Vanderbilt University School of Medicine, Nashville, TN 37232-6848, USA

652 PRB, 2220 Pierce Avenue, Departments of Biochemistry, Vanderbilt University School of Medicine, Nashville, TN 37232-6848, USA

Abstract

Background

High-throughput RNA interference (RNAi) screens have been used to find genes that, when silenced, result in sensitivity to certain chemotherapy drugs. Researchers therefore can further identify drug-sensitive targets and novel drug combinations that sensitize cancer cells to chemotherapeutic drugs. Considerable uncertainty exists about the efficiency and accuracy of statistical approaches used for RNAi hit selection in drug sensitivity studies. Researchers require statistical methods suitable for analyzing high-throughput RNAi screening data that will reduce false-positive and false-negative rates.

Results

In this study, we carried out a simulation study to evaluate four types of statistical approaches (fold-change/ratio, parametric tests/statistics, sensitivity index, and linear models) with different scenarios of RNAi screenings for drug sensitivity studies. With the simulated datasets, the linear model resulted in significantly lower false-negative and false-positive rates. Based on the results of the simulation study, we then make recommendations of statistical analysis methods for high-throughput RNAi screening data in different scenarios. We assessed promising methods using real data from a loss-of-function RNAi screen to identify hits that modulate paclitaxel sensitivity in breast cancer cells. High-confidence hits with specific inhibitors were further analyzed for their ability to inhibit breast cancer cell growth. Our analysis identified a number of gene targets with inhibitors known to enhance paclitaxel sensitivity, suggesting other genes identified may merit further investigation.

Conclusions

RNAi screening can identify druggable targets and novel drug combinations that can sensitize cancer cells to chemotherapeutic drugs. However, applying an inappropriate statistical method or model to the RNAi screening data will result in decreased power to detect the true hits and increase false positive and false negative rates, leading researchers to draw incorrect conclusions. In this paper, we make recommendations to enable more objective selection of statistical analysis methods for high-throughput RNAi screening data.

Background

Over the last decade, short RNA molecules (~20 to 30 nt) have emerged as critical regulators of the expression and function of eukaryotic genes. In particular, RNA interference (RNAi) is a valuable tool for modulating gene expression through the introduction of short interfering RNAs, including small interfering RNAs (siRNAs) and short hairpin RNAs (shRNAs)

Large-scale systematic RNAi screens aim to test hundreds, or even thousands, of siRNAs/shRNAs to identify hits rapidly and accurately. One major challenge of data processing and analysis for siRNA or shRNA screens in cancer research is to identify efficiently and accurately genes that, when lost, significantly reduce or increase cell growth/viability in response to chemical treatment. Two types of error can occur with screening experiments: false-positives and false-negatives. Strategies to reduce false-positives and false-negatives in the laboratory setting focus on making technical and procedural improvements and increasing the number of replicate measurements. It is also important to realize that enhanced statistical analysis methods also play an essential role in reducing error.

A number of statistical approaches have been applied to the analysis of high-throughput RNAi data. In their application, however, it is unclear whether: (1) effects of both the drug and the RNAi, as well as their interaction effect, are taken into consideration; (2) quantitative variation between and within replicates is taken into account in the estimation; and (3) decision error rates false-positive and false-negative are appropriately controlled. In this study, we carried out a simulation study to evaluate and compare statistical approaches for using RNAi screens to identify genes that alter sensitivity to chemotherapeutic drugs. We focused on combined RNAi and drug effect on cell viability, control of false-positive and false-negative rates, and the influence of drug concentration on the statistical power. The methods being evaluated were also applied to a real loss-of-function RNAi screening dataset to identify genes that modulate paclitaxel sensitivity in breast cancer cells.

Methods

Data processing and normalization

Several sources of noise, including technical and procedural factors, may influence measurement quality, generating inferential errors. Usually normalization is done prior to data analysis in RNAi screening studies such that variations contributed by unequal amounts of cells and/or RNAi are substantially reduced. Within-plate normalization can be performed using the non-silencing RNAi controls in the plate as a reference to give a relative measurement of target-gene knockdown effect, often adjusting for the variance by dividing by the standard deviation (SD) or median absolute deviation (MAD). Some approaches use a positive control or both positive and negative controls

Statistical approaches

One major purpose of RNAi screens is to identify genes that mediate the effects on cells of certain conditions, such as treatment with a chemotherapeutic drug or endocrine therapy. Such experiments explore the effect of gene knockdown in treated versus untreated wells, aiming to find meaningful associations between genes and the treatment. Different rules have been used to identify hits. A commonly used parametric approach is mean ± kSD under the assumption of normality

Under the assumption that most features are not significant in high-dimensional data analysis, feature selection methods like Lasso (L_{1 }penalty) and Elastic Nets (both L_{1 }and L_{2 }penalty) and their variants are often found useful and efficient in dimension reduction and feature selection. However, it may be difficult to adopt similar strategies to RNAi screening studies for drug sensitivity evaluation because, firstly, our main interest focuses on testing the gene-drug interactions, therefore in addition to siRNAs whose gene-drug interaction effect showed significance, the final model also needs to include drug effect regardless of its statistical significance. This however cannot be guaranteed by the automatic variable selection methods mentioned above. Secondly, when the number of features (

Based on above considerations, we conducted a simulation study to evaluate the performance of commonly applied methods: fold change,

Fold change/ratio

Fold change is the most intuitive approach used to represent the relative cell viability between two conditions, which usually is calculated as average cell viability (measured, for example, through an Alamar blue dye assay) over all wells in a condition divided by average viability over all wells in another condition. Because most genes knocked down in a siRNA screen do not have a significant effect on cell viability/growth in the background of the treatment, the log2 viability ratios between treated and untreated wells will be around zero for most genes. An arbitrary cut-off level, such as two- or three-fold change, is typically used to select hits for further experiments and analysis.

Parametric tests/statistics

Many biologists favor tests of two-group comparison for their easy calculation and interpretation. One widely used test is Student's two-sample

Sensitivity index (SI)

The SI method was developed to measure the influence of siRNA-induced gene knockdown on drug sensitivity, by estimating the difference between the expected and observed combined effects of RNAi and drug on cell viability. Different from the methods discussed above, the SI method estimates both the influence of siRNA-induced gene knockdown on drug sensitivity and the individual drug and RNAi effects. The SI index can be calculated for each siRNA as SI = (Rc/Cc*Cd/Cc)-(Rd/Cc), where Rc is the average viability in drug-untreated wells transfected with active siRNA, Rd is the average viability in drug-treated wells with active siRNA, Cc is the average viability in drug-untreated wells with control siRNA, and Cd is the average viability in drug-treated wells with control siRNA. The SI value ranges from -1 to 1, with positive values indicating a sensitizing effect and negative values indicating an antagonizing effect. The SI method makes no assumption about the underlying probability distribution and therefore no

Linear models with an siRNA-drug interaction effect

The SI method attempts to estimate combined RNA and drug effect. Nevertheless, one major disadvantage of the SI method is that it ignores the cross-plate variation of a particular siRNA, as the calculation of sensitivity ratio (Rd/Cd)/(Rc/Cc) involves only averaged reading levels over the replicate plates. Model-based methods are often used for feature selection in other types of high-throughput genomic data, including gene-expression microarray data and single nucleotide polymorphism (SNP) data. In our study, we used a simple linear model with an interaction term to assess RNAi effect, drug effect, and their combined effect. Assuming normal distribution, a full linear model _{2 }of cell viability (Y) for each siRNA _{1i
}, yes/no), RNAi effect (_{2i
}, yes/no), and their interaction term (_{1i
}
_{2i
}).

This model not only allows for estimating the gene-drug effect but also takes into consideration the variance among the replicates in its estimations. A test based on the difference between the deviance of the null model (model with no explanatory variables) _{0 }and the deviance of the fitted full model _{2 }may yield significant result when the drug effect is significant, even if the siRNA does not have any effect on cell viability. Therefore, we calculated the difference between the residual deviance of the fitted full model _{2 }and the deviance of the reduced model _{1 }including only drug effect (_{1}):

This statistic, _{1 }-_{2}, follows a chi-square distribution with 2 degrees of freedom. The _{2}) in _{1 }is that a significant RNAi effect alone without a significant interaction effect with drug treatment also provides vital information about the gene that is silenced, which can be very useful in identifying novel therapeutic targets for future studies.

Simulation of datasets

We evaluated the methods using datasets simulated to represent different scenarios corresponding to a given combination of parameters of number of true hits, the amount of noise, the skewness of the data, the strength of chemotherapeutic drug effect, and the RNAi effect. We focused on combined RNAi and drug effect on cell viability, control of false-positive and false-negative rates, and the influence of drug concentration on the statistical power.

Data for ten 96-well plates with three, six, nine, or twelve replicates were simulated. For each scenario, 500 simulations were carried out. For each simulation, a number of true hits were drawn randomly from the distribution Uniform{10, 11, ..., 60} with an average of 35 out of 900 siRNA wells being truly sensitizing or antagonizing. The viability measurements of the samples transfected with non-hits were generated from N(_{NH}
^{2}), with _{
NH*}
_{
1
}, ^{2}
_{1 }
_{
NH*}
_{
2
}, ^{2}
_{2 }
_{1 }
_{2 }
_{1 }
_{2 }
_{1 }
_{2 }
_{1 }
_{2 }
_{trt }
_{
unt}*_{
unt
}= _{
NH
}, _{
NH*}
_{1 }
_{
NH*}
_{2 }
_{
NH*}
_{ctl }
_{rna}
_{rna }
_{
NH
}or _{
NH*}
_{NH}
_{1}
_{2}
_{1 }
_{2 }

To evaluate the robustness of the methods for skewed data, gamma distributions were used instead of normal. The shape and scale parameters of gamma distributions were calculated by solving ^{2 }= rλ^{2 }

Criteria for the evaluation of statistical approaches

In practice, RNAi screening studies often involve a great deal of variation and noise in the raw data. Moreover, due to cost constraints, the number of replicates is often very limited. Hypothesis-testing under such conditions is, therefore, error-prone, with errors falling into two types: type I error (false-positive, FP) and type II error (false-negative, FN). To evaluate the performance of the methods, we calculated the false-positive rate (FPR) and the false-negative rate (FNR) of each method and in each scenario:

The FPR corresponds to the portion of genes that, when silenced, have no influence on drug sensitivity among those identified as influential by the method. The FNR corresponds to the portion of genes influencing drug sensitivity among those claimed non-influential by the method.

Real data analysis

Paclitaxel is a potent anti-microtubule agent used in the treatment of patients with locally advanced and metastatic breast cancer. Despite its wide use, paclitaxel-based chemotherapy results in full response in only a small portion of patients; many patients have an incomplete response or are resistant to treatment. We performed a loss-of-function RNAi screen to identify genes that modulate paclitaxel sensitivity

Results

Simulation study

We report nine most representative scenarios simulated separately for each of the three, six, nine, and twelve-replicate datasets as described above. Because no critical value/threshold can be universally applied to all methods, results based on significance thresholds of different methods are not directly comparable. For the purpose of fair comparison, we selected the same number of hits from each method according to the true number of hits simulated in each dataset. We ranked all genes based on their significance assessed by each method and selected the top _{TH }
_{TH }=

We compared the accuracy of the methods at different combinations of level of noise, drug effect, and RNAi effect. Table ^{2}

Recommendation of analysis method based on simulation study

**Number of replicates**

**Scenario**

**Noise**

**Drug effect**

**RNA effect**

**Performance ranking order***

3

1

low

strong

strong

LM,

2

moderate

strong

strong

LM,

3

strong

strong

strong

LM,

4

low

moderate

strong

LM,

5

low

weak

strong

LM,

6

moderate

weak

strong

LM,

7

strong

weak

strong

LM, SI,

8

low

strong

moderate

LM,

9

low

strong

weak

LM,

6

1

low

strong

strong

LM,

2

moderate

strong

strong

LM,

3

strong

strong

strong

LM,

4

low

moderate

strong

LM,

5

low

weak

strong

LM,

6

moderate

weak

strong

LM,

7

strong

weak

strong

LM,

8

low

strong

moderate

LM,

9

low

strong

weak

LM,

9

1

low

strong

strong

LM,

2

moderate

strong

strong

LM,

3

strong

strong

strong

LM, SI,

4

low

moderate

strong

LM,

5

low

weak

strong

LM,

6

moderate

weak

strong

LM,

7

strong

weak

strong

LM,

8

low

strong

moderate

LM,

9

low

strong

weak

LM,

12

1

low

strong

strong

LM,

2

moderate

strong

strong

LM, SI,

3

strong

strong

strong

LM, SI,

4

low

moderate

strong

LM,

5

low

weak

strong

LM,

6

moderate

weak

strong

LM,

7

strong

weak

strong

LM,

8

low

strong

moderate

LM,

9

low

strong

weak

LM,

* The ranking is based on average FNRs and FPRs of the methods: linear model (LM),

Since power = sensitivity = 1-FNR, controlling FNR automatically controls power/sensitivity. Figures

Scenario 1

**Scenario 1**. Low noise, high drug effect, high RNA effect.

Scenario 2

**Scenario 2**. Moderate noise, high drug effect, high RNA effect.

Scenario 3

**Scenario 3**. High noise, high drug effect, high RNA effect.

Scenario 4

**Scenario 4**. Low noise, moderate drug effect, high RNA effect.

Scenario 5

**Scenario 5**. Low noise, weak drug effect, high RNA effect.

Scenario 6

**Scenario 6**. Moderate noise, weak drug effect, high RNA effect.

Scenario 7

**Scenario 7**. High noise, weak drug effect, high RNA effect.

Scenario 8

**Scenario 8**. Low noise, high drug effect, moderate RNA effect.

Scenario 9

**Scenario 9**. Low noise, high drug effect, weak RNA effect.

When a strong drug effect is present, the SI method is less powerful with skewed data than with unskewed data. The LM method is very stable: FNR with skewed data is only slightly higher than with unskewed data in simulations with a small number of replicates. The

Our simulation study suggests that the LM method performs overwhelmingly better than all other methods considered. When the data have a strong drug and RNAi effect but with a small number of replicates, the

Hits from shRNA/siRNA screening

After normalization, the SI method was initially applied to identify gene hits, as this method had recently been proposed and published as an approach to account for RNAi-drug interaction. The SI score was calculated for each of the shRNAs and siRNAs. Genes were then ranked according to the SI score, and the top hits for each cell line were selected for further analysis. After the simulation study described above was completed, we subsequently applied the t-test, fold-change, and LM methods to the same data. The top hits selected by SI also ranked very high on the list generated by LM, though a small number of mismatches were observed (for example, SP1 was identified as a top hit only in the MDA-MB-468 cell line with the SI method, but ranked high in both cell lines with the LM method). This is expected because the data has strong drug and RNAi effects; also we only validated top hits with the strongest combined effect.

FRAP1 (mTOR) was a hit in both cell lines, as anticipated. This gene is a known target for enhancing paclitaxel sensitivity and was used as a positive control in each plate of our screen to allow for cross-plate comparisons of drug sensitivity

PPM1D, SP1, and TGF-β1 were hits of particular interest, as these genes encode proteins with known chemical inhibitors, which could be tested in combination with paclitaxel for biological effect. When the chemical inhibitors CCT007093 (PPM1D inhibitor) and mithramycin (SP1-binding inhibitor) were used in combination with paclitaxel, we observed synergistic growth inhibition of breast cancer cell cultures. We observed similar results with the transforming growth factor beta receptor inhibitor, LY2109761, which targets the TGF-β1 signaling pathway; LY2109761 plus paclitaxel synergistically inhbited growth of breast cancer cell lines in 3D culture. These examples provide strong evidence that validate our identified druggable gene targets that modulate paclitaxel sensitivity.

Discussion

It is typical that high-throughput RNAi screening studies attempt to test hundreds or thousands of siRNAs with a relatively small number of replicates for each. It has been brought to our attention that popular statistical approaches for analyzing such data, although commonly applied, appear to have drawbacks in terms of efficiency and accuracy. Using simulated datasets for different scenarios, we evaluated and compared these approaches to the linear model we conducted for estimating both the influence of siRNA-induced gene knockdown on drug sensitivity and the individual effects of the drug and siRNA on cell viability. Overall, the LM method outperforms other evaluated methods mainly because it not only takes the variation among replicate measurements but also focuses on estimating the combined effect of drug and RNAi by incorporating an interaction term in the model.

Because a relatively small number of true hits (10 to 60) in our study were simulated among 900 siRNAs, the numbers of both false-negatives and true-positives are small relative to the total number of siRNAs, making the FNR very sensitive to the total number of true-positives. On the other hand, the number of true-negatives is large relative to the number of false-positives; therefore FPRs are, in general, very low (< 5%).

The

As previously mentioned, in practice, low drug effect usually results from low drug concentration. Interestingly, recent studies have identified targets that sensitize cancer cells to chemotherapy drugs of a much lower concentration than otherwise required, such as paclitaxel for non-small-cell lung cancer cells

Conclusions

RNAi screening can identify genes that mediate sensitivity or resistance to certain chemotherapeutic drugs and novel drug combinations that can sensitize cancer cells to a chemotherapeutic drug. However, applying an inappropriate statistical method or model to RNAi screening data will result in decreased power to detect true hits, increase the false-positive and false-negative rates, and consequently lead to incorrect conclusions. Based on the results of our simulation study, the authors have made recommendations to enable objective selection of statistical analysis methods for high-throughput RNAi screening data.

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

FY performed the simulations, applied the various methodologies for analyzing RNAi screening data, and performed the statistical analyses to compare these methodologies. JAB performed the experiments to generate the real-world breast cancer datasets, and JAP supervised this laboratory research. YS guided and supervised the overall project. All authors contributed to writing of the manuscript.

Acknowledgements

The authors would like to thank Lynne Berry and Peggy Schuyler for editorial advice. This work was supported by the following grants: Cancer Center Support Grant (P30 CA068485), Vanderbilt Molecular Target Discovery and Development Center (RC2 CA148375), and Vanderbilt Special Program of Research Excellence in Breast Cancer (P50 CA098131).

This article has been published as part of