Center for Health Informatics and Bioinformatics, New York University Langone Medical Center, New York, NY 10016, USA
Department of Medicine, Division of Translational Medicine, New York University School of Medicine, New York, NY 10016, USA
Department of Pathology, New York University School of Medicine, New York, NY 10016, USA
Department of Biostatistics, Vanderbilt University, Nashville, TN, 37232, USA
Abstract
Background
The discovery of molecular pathways is a challenging problem and its solution relies on the identification of causal molecular interactions in genomics data. Causal molecular interactions can be discovered using randomized experiments; however such experiments are often costly, infeasible, or unethical. Fortunately, algorithms that infer causal interactions from observational data have been in development for decades, predominantly in the quantitative sciences, and many of them have recently been applied to genomics data. While these algorithms can infer unoriented causal interactions between involved molecular variables (i.e., without specifying which one is the cause and which one is the effect), causally orienting all inferred molecular interactions was assumed to be an unsolvable problem until recently. In this work, we use transcription factortarget gene regulatory interactions in three different organisms to evaluate a new family of methods that, given observational data for just two causally related variables, can determine which one is the cause and which one is the effect.
Results
We have found that a particular family of causal orientation methods (IGCI Gaussian) is often able to accurately infer directionality of causal interactions, and that these methods usually outperform other causal orientation techniques. We also introduced a novel ensemble technique for causal orientation that combines decisions of individual causal orientation methods. The ensemble method was found to be more accurate than any best individual causal orientation method in the tested data.
Conclusions
This work represents a first step towards establishing context for practical use of causal orientation methods in the genomics domain. We have found that some causal orientation methodologies yield accurate predictions of causal orientation in genomics data, and we have improved on this capability with a novel ensemble method. Our results suggest that these methods have the potential to facilitate reconstruction of molecular pathways by minimizing the number of required randomized experiments to find causal directionality and by avoiding experiments that are infeasible and/or unethical.
Background
The discovery of molecular pathways that drive diseases and vital cellular functions is a fundamental activity of biomedical research. Unraveling disease pathways is a major component in the efforts to develop new therapies that will effectively fight deadly diseases. Furthermore, knowing pathways significantly facilitates the design of personalized medicine modalities for diagnosis, prognosis, and management of diseases. The discovery of pathways is a challenging problem and its solution to a large extent relies on the identification of
By causal molecular interactions or relations we mean interactions of molecular variables that match the notion of randomized controlled experiment, which is the de facto standard for assessing causation in the general sciences and biomedicine
Causal molecular interactions can be discovered using randomized experiments such as interference with RNA (e.g., shRNA, siRNA); however such experiments are often costly, infeasible, or unethical. Fortunately, over the last 20 years many algorithms that infer causal interactions from
In our prior work we evaluated the ability of stateoftheart causal discovery algorithms to denovo identify
Over the last 5 years researchers have developed a new class of methods that, given observational data for just two causally related variables X and Y, aim to determine which variable is the cause and which one is the effect (e.g., separate X→Y from X←Y)
As promising as these new causal orientation methods are, they have not been previously applied in genomics, where the data is usually noisy and the sample sizes are relatively small compared to prior test applications of these methods
Methods
Causal orientation methods
As mentioned above, the purpose of the tested causal orientation methods is to separate cause from effect given data for just two variables X and Y
While each causal orientation method has its own principles and sufficient assumptions that are outlined in Table
Method
Reference
Key principles
Sufficient assumptions for causally orienting X → Y
Sound
ANM
Assuming X → Y with Y = f(X) + e_{1}, where X and e_{1 }are independent, there will be no such additive noise model in the opposite direction X ← Y, X = g(Y) + e_{2}, with Y and e_{2 }independent.
• Y = f(X) + e_{1};
• X and e_{1 }are independent;
• f is nonlinear, or one of X and e is nonGaussian;
• Probability densities are strictly positive;
• All functions (including densities) are 3 times differentiable.
Yes
PNL
Assuming X → Y with Y = f_{2}(f_{1}(X) + e_{1}), there will be no such model in the opposite direction X←Y, X = g_{2}(g_{1}(Y) + e_{2}) with Y and e_{2 }independent.
• Y = f_{2}(f_{1}(X) + e_{1});
• X and e_{1 }are independent;
• Either f_{1 }or e_{1 }is Gaussian;
• Both f_{1 }and f_{2 }are continuous and invertible.
Yes
IGCI
Assuming X→Y with Y = f(X), one can show that the KLdivergence (a measure of the difference between two probability distributions) between P(Y) and a reference distribution (e.g., Gaussian or uniform) is greater than the KLdivergence between P(X) and the same reference distribution.
• Y = f(X) (i.e., there is no noise in the model);
• f is continuous and invertible;
• Logarithm of the derivative of f and P(X) are not correlated.
Yes
GPIMML
Assuming X→Y, the least complex description of P(X, Y) is given by separate descriptions of P(X) and P(YX). By estimating the latter two quantities using methods that favor functions and distributions of low complexity, the likelihood of the observed data given X→Y is inversely related to the complexity of P(X) and P(Y  X).
• Y = f(X, e);
• X and e are independent;
• e is Gaussian;
• The prior on f and P(X) factorizes.
No
ANMMML
Same as for GPIMML, except for a different way of estimating P(Y  X) and P(X  Y).
• Y = f(X) + e;
• X and e are independent;
• e is Gaussian.
• The prior on f and P(X) factorizes.
No
GPI
Assuming X→Y with Y = f(X,e_{1}), where X and e_{1 }are independent and f is "sufficiently simple", there will be no such model in the opposite direction X←Y, X = g(Y,e_{2}) with Y and e_{2 }independent and g "sufficiently simple".
Same as for GPIMML.
No
ANMGAUSS
Same as for ANMMML, except for the different way of estimating P(X) and P(Y).
Same as for ANMMML.
No
LINGAM
Assuming X→Y, if we fit linear models Y = b_{2}X+e_{1 }and X = b_{1}Y+e_{2 }with e_{1 }and e_{2 }independent, then we will have b_{1 }< b_{2}.
• Y = b_{2}X+e_{1};
• X and e_{1 }are independent;
• e_{1 }is nonGaussian.
Yes
The last column indicates whether a method is sound, i.e. it can provably orient a causal structure under its sufficient assumptions. Because causal orientation methodologies are fairly new and not completely characterized, it is possible that proofs of correctness will become available for GPIMML, ANMMML, GPI, and ANMGAUSS. All methods implicitly assume that there are no feedback loops. The noise term in the models is denoted by small "e".
This file contains (1) brief description of causal orientation algorithms; (2) results of causal orientation methods ANM, PNL, and GPI obtained by assessing statistical significance of the forward and backward causal models; (3) detailed results of significance testing of IGCI Gaussian/Entropy and Gaussian/Integral methods; (4) explanation of performance increase due to adding small amount of noise or reducing the sample size in YEAST gold standard.
Click here for file
The majority of tested causal orientation methods (IGCI, LINGAM, GPIMML, ANMMML, ANMGAUSS) output two scores indicating likelihood of the forward causal model (X → Y) and the backward one (X ← Y). Other tested methods (ANM, PNL, GPI) output two pvalues indicating significance of the forward and backward causal models. In order to make all methods directly comparable to each other, we decided to force them to make causal orientation decisions for all tested causal interactions. This was achieved by comparing scores or pvalues of the forward and backward causal models and selecting the most likely orientation. This approach follows the practices of previously published applications of causal orientation methods by their inventors
We also note that an alternative approach for the ANM, PNL, and GPI methods is to select a model (forward/backward) that is statistically significant at a given alpha level. The latter approach can sometimes improve accuracy of the causal orientation method while reducing the fraction of causally oriented edges
Finally, prior to application of the causal orientation methods, we standardized the data to mean zero and standard deviation one.
Gold standard construction and observational data
The primary challenges in evaluating causal orientation methods for genomics applications are (i) limited availability of known gold standards of causal molecular interactions and (ii) limited sample sizes of the available observational data. To overcome these challenges we focused on transcription factorgene regulatory interactions that have been discovered on the genome wide level and experimentally verified in model organisms
We used the following four gold standards: (i) interactions of the NOTCH1 transcription factor and its target genes in human Tcell acute lymphoblastic leukaemia (denoted as NOTCH1); (ii) interactions of the RELA transcription factor and its target genes in human Tcell acute lymphoblastic leukaemia (denoted as RELA); (iii) interactions of 140 transcription factors and their target genes in Escherichia coli (denoted as ECOLI); and (iv) interactions of 115 transcription factors and their target genes in Saccharomyces cerevisiae (denoted as YEAST). We used microarray gene expression data from the public domain for orientation of causal relations in each gold standard. The summary statistics of gold standards and corresponding microarray gene expression datasets are given in Table
Task
name
Reference/source
# TFs in GS
# genes in GS
# gene probes for GS genes in gene expression data
# TFgene interactions
# TFgene interactions significant at FDR = 0.05
140
913
913
1,885
115
1,834
1,834
3,541
1
302
813
813
1
1,420
3,657
3,657
"TF" stands for "transcription factor". Statistically significant associations were determined using Fisher's Ztest at 5% FDR in microarray gene expression data (please see text for details).
Task name
Reference/source
# samples
907
530
174
174
Only TALL samples were selected for NOTCH1 and RELA in order to match cell population used for creation of the respective gold standard.
Once each of the gold standards was constructed, we removed interactions without statistically significant associations (in the observational data) according to Fisher's Ztest
All gold standards and microarray gene expression datasets are available for download from
Functional gene expression data was first used to identify genes that are downstream (but not necessarily directly) of a particular transcription factor. The samples in such data are randomized to either 'experiment' (e. g., siRNA knockdown of the transcription factor of interest) or 'control' treatment. All genes that are differentially expressed between 'experiment' and 'control' treatments are expected to be downstream of the transcription factor. We have used a ttest with α = 0.05 to identify such genes.
Genomewide binding data (ChIPonchip for NOTCH1 and ChIPsequencing for RELA) was then employed to identify direct binding targets of each transcription factor. Specifically, for each studied transcription factor we obtained the set of genes with detected promoter regiontranscription factor binding according to the primary study that generated binding data.
We note that using genomewide binding data by itself is insufficient to find downstream functional targets of a transcription factor, because many binding sites can be nonfunctional
The Saccharomyces cerevisiae (denoted as YEAST) gold standard was built by identifying the promoter sequences that are both bound by transcription factors according to ChIPonchip data at 0.001 alpha level and conserved within 2 related species in the Saccharomyces genus
The Escherichia coli (denoted as ECOLI) gold standard was constructed from RegulonDB (version 6.4), a manually curated database of regulatory interactions obtained mainly through a literature search
The gold standards YEAST and ECOLI contain relations of the type "transcription factor → gene" and "transcription factor → transcription factor" (where "gene" refers to a target gene that is not a transcription factor). We decided to simplify the setting of our evaluation when we assess whether the inferred causal orientation X→Y or X←Y is correct, and restricted attention to only interactions of the type "transcription factor → gene". This results in minimizing the number of cases with feedback that can be represented by causal edges in both directions. Note that it is not currently possible to comprehensively apply this filtering step to NOTCH1 and RELA gold standards because the transcription factors are not well known in human cells.
Performance metrics and statistical significance testing
Two metrics were used to assess performance of causal orientation algorithms. The first metric is accuracy which is the percentage of causal interactions that have been oriented correctly. A method that orients all causal interactions in the gold standard as "transcription factor → gene" would achieve an accuracy of 1; a method that orients all interactions as "gene → transcription factor" would achieve an accuracy of 0; and a method that flips a fair coin to make every orientation decision would on average achieve an accuracy of 0.5.
The second metric is area under ROC curve (AUC), which is known to be more discriminative than accuracy because it takes into account the confidence of orientation decisions
a)
b)
Variable 1
Causal Edge
Variable 2
Variable 1
Causal Edge
Variable 2
Response
NOTCH1
→
ABCF2
NOTCH1
→
ABCF2
+
NOTCH1
→
EIF4E
EIF4E
←
NOTCH1

NOTCH1
→
SFRS3
NOTCH1
→
SFRS3
+
NOTCH1
→
NUP98
NUP98
←
NOTCH1

NOTCH1
→
CYCS
NOTCH1
→
CYCS
+
NOTCH1
→
ZNHIT
ZNHIT
←
NOTCH1

NOTCH1
→
ATM
NOTCH1
→
ATM
+
NOTCH1
→
TIMM9
TIMM9
←
NOTCH1

A fragment of the gold standard is shown in a). The edges always point from a transcription factor (NOTCH1) to its target gene. 50% of the edges are represented as "transcription factor → gene" and the other 50% as "gene ← transcription factor" in b). This constructs a response variable with positives corresponding to "→" edges (shown in black) and negatives corresponding to "←" edges (shown in red).
Given values of the above two performance metrics (accuracy and AUC), we need to assess their statistical significance, i.e. find out if the causal orientation is better than by chance. Notice that our gold standards are such that many causal edges are not independent because they share the same transcription factor. That is why we chose to apply an exact statistical testing procedure that can accurately estimate the variance of orientation by chance in our setting
Methodology for sensitivity analyses
In order to study sensitivity to sample size (number of observations), we sample without replacement from the original gene expression data nested subsets of size 10, 20, 30, ..., N, where N is the number of samples in the dataset. Specifically, the subset of size 10 is included in the subset of size 20, which is in turn included in the subset of size 30, and so on. We then run the causal orientation algorithms on each subset and compute performance. This process is repeated with different sampled nested subsets, and mean performance and variance are estimated over all runs. For the NOTCH1 and RELA gold standards, we used 100 subsets of each size, while for the more computationally expensive YEAST and ECOLI gold standards we used 20 subsets of each size.
For the sensitivity analysis to noise, we add a certain proportion (p) of random Gaussian noise to the gene expression data for both transcription factors and their target genes, run causal orientation methods in the noisy data, report their performance, and repeat the entire process to assess variance (again, 100 times for NOTCH1 and RELA and 20 times for YEAST and ECOLI). Denoting by X the transcription factor and by Y its target gene, the noisy transcription factor X′ and gene Y′ are defined as follows: X′ = (1p) · X + p · N(M_{X},S_{X}) and Y′ = (1p) · Y + p · N(M_{Y},S_{Y}), where N(m,s) is a Normally distributed random variable with mean m and standard deviation s, and M_{X}, M_{Y }and S_{X}, S_{Y }are means and standard deviations of X and Y in the original data (prior to noise addition). We use the following proportions of noise (values of p): 0.05, 0.10, 0.15, ..., 0.90, 0.95, 1.00.
A new ensemble method for causal orientation
As an enhancement to using individual causal orientation techniques, we introduce ensemble causal orientation models that combine decisions of all available individual causal orientation methods in order to produce a more powerful predictor of causal orientation. These methodologies are popular in the field of supervised learning, where nonrandom weak learners are often combined to produce a more accurate predictor
In this study we experimented with a straightforward approach to ensemble modelling, where we train a logistic regression model
As with every supervised learning procedure that is trained and tested using the same dataset, it is essential to split the available data into nonoverlapping training and testing sets, whereby the training set is used to fit a learning model and the testing set is used to estimate its performance
Finally, in addition to exploring holdout validation performance of the ensemble models, we trained and tested the ensemble models on different gold standards. In practice this approach can be justified if the data distributions in the gold standards used for training and testing of ensemble models are similar. It also resembles a practical situation when a gold standard is known in a previously studied dataset but is not known in a new but distributionally similar one.
Results
Evaluating causal orientation methods with the accuracy metric
The causal orientation accuracy values are given in Table
Method
ECOLI
YEAST
NOTCH1
RELA
ANM
0.462
0.383
0.476
0.396
PNL
0.453
0.471
0.521
0.520
IGCI (Uniform/Entropy)
0.647
0.427
0.611
0.692
IGCI (Uniform/Integral)
0.605
0.441
0.561
0.669
IGCI (Gaussian/Entropy)
0.742
0.555
0.848
0.898
IGCI (Gaussian/Integral)
0.645
0.587
0.729
0.835
GPIMML
0.485
0.390
0.251
0.395
ANMMML
0.428
0.316
0.183
0.172
GPI
0.526
0.401
0.548
0.506
ANMGAUSS
0.480
0.483
0.727
0.462
LINGAM
0.469
0.451
0.367
0.387
RANDOM
0.500
0.500
0.500
0.500
For each gold standard (column) dark orange cells correspond to methods that have high values of accuracy, while white cells correspond to methods that have low values of accuracy. Accuracies higher than 0.50 are shown in bold.
Method
ECOLI
YEAST
NOTCH1
RELA
ANM




PNL


5
5
IGCI (Uniform/Entropy)
2

3
3
IGCI (Uniform/Integral)
3

4
4
IGCI (Gaussian/Entropy)
1
2
1
1
IGCI (Gaussian/Integral)
2
1
2
2
GPIMML




ANMMML




GPI
4

4
5
ANMGAUSS


2

LINGAM




Ranks were computed only for the methods with accuracies greater than 0.50. The lower the rank, the better the accuracy of the causal orientation method for the given gold standard. The computation of rank took into account statistical variability, e.g. accuracies 0.647 and 0.645 obtained by the two IGCI methods in the ECOLI gold standard are statistically indistinguishable; that is why they have the same rank value.
As can be seen, IGCI Gaussian/Entropy and IGCI Gaussian/Integral methods achieve the highest accuracies in each of the four gold standards. In general, the other causal orientation methods perform worse, and some methods (e.g., ANMMML) consistently prefer wrong decisions and have accuracies lower than 0.5.
Interestingly, if we consider the best performing method (IGCI Gaussian/Entropy) with the average rank 1.25, its results are statistically significant at alpha = 0.05 according to the exact test (described in the Methods section) only for the ECOLI gold standard (pvalue < 0.001). The second best performing method (IGCI Gaussian/Integral) with the average rank 1.75 achieves significance in two out of four gold standards (pvalue < 0.001 for ECOLI and 0.003 for YEAST) at alpha = 0.05. The reason why the IGCI Gaussian/Integral method achieves significance in more gold standards than the best performing technique IGCI Gaussian/Entropy is the small variance of the former method. The detailed statistical significance results including null distributions are given in the Additional file
The superior and often statistically significant performance of the two IGCI methods compared to other techniques was a surprising finding that we did not expect theoretically. IGCI assumes a noise free model (Table
Evaluating causal orientation methods with the AUC metric
The causal orientation AUC values are given in Table
Method
ECOLI
YEAST
NOTCH1
RELA
ANM
0.464
0.379
0.456
0.369
PNL
0.443
0.464
0.520
0.520
IGCI (Uniform/Entropy)
0.713
0.409
0.708
0.805
IGCI (Uniform/Integral)
0.642
0.437
0.631
0.757
IGCI (Gaussian/Entropy)
0.813
0.613
0.935
0.967
IGCI (Gaussian/Integral)
0.724
0.655
0.834
0.927
GPIMML
0.488
0.370
0.184
0.333
ANMMML
0.393
0.237
0.078
0.071
GPI
0.536
0.396
0.594
0.513
ANMGAUSS
0.474
0.476
0.807
0.446
LINGAM
0.462
0.463
0.362
0.392
RANDOM
0.500
0.500
0.500
0.500
For each gold standard (column) dark orange cells correspond to methods that have high values of AUC, while white cells correspond to methods that have low values of AUC. AUCs higher than 0.50 are shown in bold.
Method
ECOLI
YEAST
NOTCH1
RELA
ANM




PNL


5
5
IGCI (Uniform/Entropy)
2

3
3
IGCI (Uniform/Integral)
3

4
4
IGCI (Gaussian/Entropy)
1
2
1
1
IGCI (Gaussian/Integral)
2
1
2
2
GPIMML




ANMMML




GPI
4

4
5
ANMGAUSS


2

LINGAM




Ranks were computed only for the methods with AUCs greater than 0.50. The lower the rank, the better the AUC of the causal orientation method for the given gold standard. The computation of rank took into account statistical variability, e.g. the AUCs of 0.724 and 0.713 obtained by the two IGCI methods in the ECOLI gold standard are statistically indistinguishable; that is why they have the same rank value.
Similarly to the accuracy results, IGCI Gaussian/Entropy and IGCI Gaussian/Integral methods achieve the highest AUCs in each of the four gold standards. Other causal orientation methods perform worse, and some methods (e.g., ANMMML) consistently prefer wrong decisions and have AUCs lower than 0.5.
The statistical significance analysis of IGCI Gaussian/Entropy and IGCI Gaussian/Integral is described in detail in the Additional file
Sensitivity analysis to noise
The results of sensitivity analysis to noise for the two best performing methods (IGCI Gaussian/Entropy and IGCI Gaussian/Integral) are given in Figures
Whereas in NOTCH1 and RELA gold standards it takes only 510% of noise to make the results statistically indistinguishable from orientation by chance, in YEAST and ECOLI gold standards the methods can tolerate much higher proportions of noise and still produce statistically significant results. This can be attributed to a larger number of transcription factors in YEAST and ECOLI gold standards, as well as larger sample sizes in the corresponding datasets which both decrease the variability of the results.
A decrease in performance upon the addition of noise is theoretically expected since IGCI assumes a noisefree model, and the addition of Gaussian noise violates its sufficient assumptions. Also, as can be seen in the figures, the IGCI Gaussian/Integral method has lower variance than the IGCI Gaussian/Entropy method. The above results are consistent with our prior findings and statistical significance testing by the exact test (see Figures S1S4 in the Additional file
Sensitivity analysis to sample size
The results of sensitivity analysis to sample size for the two best performing methods (IGCI Gaussian/Entropy and IGCI Gaussian/Integral) are given in Figures
Whereas in NOTCH1 and RELA gold standards results become statistically indistinguishable from orientation by chance when the sample size is <80100, in YEAST and ECOLI gold standards the methods yield statistically significant results for smaller sample sizes. This can be attributed to a larger number of transcription factors in YEAST and ECOLI gold standards which decreases variability of the results.
Also, as can be seen in the figures, the IGCI Gaussian/Integral method has lower variance than the IGCI Gaussian/Entropy method. The above results are consistent with our prior findings in statistical significance testing by the exact test (see Figures S1S4 in the Additional file
Ensemble causal orientation
For each gold standard, Table
ECOLI
YEAST
NOTCH1
RELA
Best individual causal orientation method (AUC)
0.828
0.658
0.926
0.970
Ensemble method (AUC)
0.837
0.822
0.984
0.992
Improvement (AUC)
Statistical significance of improvement (pvalue)
0.3407
<0.0001
0.0062
<0.0001
Bold pvalues indicate a statistically significant performance improvement by using an ensemble causal orientation. The pvalues were obtained from Delong's test for AUC comparison
Coefficients for the ensemble logistic regression model trained in the YEAST gold standard
Method (feature in the logistic regression model)
Beta
Pvalue
ANM
1.20
0.291
PNL
0.27
0.750
IGCI (Uniform/Entropy)
128.03
<0.0001
IGCI (Uniform/Integral)
135.07
<0.0001
IGCI (Gaussian/Entropy)
99.20
<0.0001
IGCI (Gaussian/Integral)
106.45
<0.0001
GPIMML
1.15
0.578
ANMMML
9.87
0.017
GPI
1.45
0.298
ANMGAUSS
0.40
0.808
LINGAM
0.11
0.963
Bold values correspond to coefficients that are statistically significant at 0.05 alpha level. We note that due to multicollinearity among the IGCI Uniform methods and among the IGCI Gaussian methods, care must be taken when interpreting the logistic regression coefficients
The above results were obtained by holdout validation where we used different portions of the same gold standard for training and testing ensemble models. We also experimented with training and testing ensemble models on different gold standards. First we experimented with the RELA and NOTCH1 gold standards that were derived from the same organism and phenotype, and thus are likely to be distributionally similar and support crossgold standard application of the ensemble model. We find that an ensemble logistic regression model trained on RELA obtains AUC = 0.996 when tested on NOTCH1, and likewise an ensemble model trained on NOTCH1 obtains AUC = 0.989 when tested on RELA. Both these results significantly improve performance over the best individual causal orientation method (IGCI Gaussian/Entropy) in both NOTCH1 and RELA gold standards (with pvalues <0.0001).
In addition, we experimented with the YEAST and ECOLI gold standards which originate from different organisms and thus are unlikely to be distributionally similar; for this reason they
Discussion
This work represents the first comprehensive effort to evaluate performance and furthermore enhance the recently introduced causal orientation methods
Even though the choice of the gold standard with transcription factorgene regulatory interactions enables this study, its practical relevance may be limited in the organisms/settings where all transcription factors have already been identified. That is why we plan to work on extending this evaluation to other types of causal molecular interactions, for example in cellular protein signaling networks
In this study we have implicitly assumed that unoriented edges (representing causal interactions between a transcription factor and its target gene without specifying which of the two genes is a transcription factor and which is its regulatory target) are given by an Oracle and we have evaluated performance of only causal orientation methods. However, in practical tasks one typically has to both discover and orient edges. Although we have previously evaluated methods for discovery of unoriented edges
Finally, we think that a fruitful area of research will be to extend this study by comparison with classical causal orientation techniques that output Markov equivalence classes of graphs (based on vstructures with constraint propagation) and thus, in general, can orient only a subset of edges in the graph
Conclusions
In this paper we have taken a first step toward practical use of recent causal orientation techniques in the genomics domain. First of all, we report results of an extensive study of causal orientation methods in genomics data that utilized 12 methods/variants to distinguish cause (transcription factor) from effect (target gene) in 5,739 causal interactions. We have found that IGCI Gaussian methods
Competing interests
The authors declare that they have no competing interests.
Authors' contributions
AS conceived the study. All authors participated in the design of the study. MH and NIL performed all computational experiments. AS and MH drafted the manuscript. All authors edited, read, and approved the final manuscript.
Acknowledgements
The authors would like to acknowledge Dominik Janzing and Joris M. Mooij, who contributed to the development of the majority of causal orientation methods used in this study, and thank them for providing (i) software implementations of causal orientation algorithms, (ii) help with stating assumptions of the tested methods, (iii) ideas about statistical significance testing approach, and (iv) feedback on other aspects of the manuscript and, in particular, interpretation of the results. The authors are also grateful to Efstratios Efstathiadis and Eric Peskin for the help with providing access and running experiments on the high performance computing facility. Finally, the authors would like to thank Ioannis Aifantis for providing experimental data for NOTCH1 that was used for the development of the corresponding gold standard. The empirical evaluation was supported in part by the grants 1R01LM01117901A1 from the National Library of Medicine and 1UL1RR029893 from the National Center for Research Resources, National Institutes of Health.
This article has been published as part of