Partners Center for Personalized Genetic Medicine, Boston, MA, 02115, USA

Graduate Program in Bioinformatics, Boston University, Boston, MA, 02215, USA

Department of Genetics, Harvard Medical School, Boston, MA, 02115, USA

Wyss Institute for Biologically Inspired Engineering, Harvard University, Boston, MA, 02115, USA

Broad Institute of MIT and Harvard, Cambridge, MA, 02142, USA

Department of Biomedical Engineering, Boston University, Boston, MA, 02215, USA

Harvard-MIT Division of Health Sciences and Technology, Boston, MA, 02139, USA

Children’s Hospital Informatics Program, Boston, MA, 02115, USA

Abstract

Background

Pre-symptomatic prediction of disease and drug response based on genetic testing is a critical component of personalized medicine. Previous work has demonstrated that the predictive capacity of genetic testing is constrained by the heritability and prevalence of the tested trait, although these constraints have only been approximated under the assumption of a normally distributed genetic risk distribution.

Results

Here, we mathematically derive the absolute limits that these factors impose on test accuracy in the absence of any distributional assumptions on risk. We present these limits in terms of the best-case receiver-operating characteristic (ROC) curve, consisting of the best-case test sensitivities and specificities, and the AUC (area under the curve) measure of accuracy. We apply our method to genetic prediction of type 2 diabetes and breast cancer, and we additionally show the best possible accuracy that can be obtained from integrated predictors, which can incorporate non-genetic features.

Conclusion

Knowledge of such limits is valuable in understanding the implications of genetic testing even before additional associations are identified.

Background

Accurate pre-symptomatic prediction of disease and drug response is a vital component of personalized medicine, which could allow for improved clinical decision-making and targeted prevention strategies, easing both the burden and costs of disease

Although the accuracy of a medical test can be measured in many ways, the concepts of sensitivity and specificity are paramount

Evidence that a bound on maximum predictive accuracy exists can be found in heritability. The heritability of a trait (in the broad-sense) is the proportion of phenotypic variation in the population that can be attributed to genetic variation; that is, it reflects the contribution of genetic factors relative to environmental ones. Narrow-sense heritability measures the corresponding quantity for additive genetic variance only, which excludes genetic effects such as dominance and epistasis. The heritability of binary phenotypes can be computed directly on the observed binary scale. However, it may also be calculated on a liability scale, where it is assumed that an individual has the binary trait if their risk exceeds a threshold. Both types of heritability can be estimated using family-based studies, such as twin studies

The impact of heritability on genetic test accuracy can be seen by examining its two extremes: a trait that has 100% heritability, such as a Mendelian trait, can be predicted with certainty from the genotype; in contrast, a trait with 0% heritability is not influenced by genetic factors, and thus genetic tests cannot produce any useful information. Previous ground-breaking works have investigated the bounds prevalence and heritability impose on predictive accuracy using simulations

Results

Common complex traits are typically the combined effect of genetic and environmental factors. Since no practical predictor can account for all factors and their interactions, clinical prediction can at best assign probabilistic risks rather than deterministic outcomes. Viewed on the population level, these risk assignments can be seen as comprising a risk distribution, which is an estimate of the population’s true risk distribution. Maximal predictive accuracy occurs when the estimated risk matches the true risk.

The prevalence and heritability of any trait restrict the set of possible genetic risk distributions. If we know the risk corresponding to each individual’s genetic profile in a large sample, then we can obtain an expression for broad-sense heritability (H^{2}) on the binary scale

where _{
i
} is individual

To mathematically derive the risk distribution that yields the best genetic prediction, we model the distribution as a histogram with equally-spaced bins located from 0 to 100% representing risk groups, where the height of each bin denotes the proportion of the population who fall into that risk group (for an example, see Figure

Example risk distribution

**Example risk distribution.** This distribution has a prevalence of 30% and a heritability of 10%. The mean of the distribution equals the prevalence of the trait.

Using this approach, we have derived the maximum limits on the genetic predictive accuracy of any binary trait given only its prevalence and heritability. These values are tabulated in Additional files

**Table of maximum AUCs.** These are the maximum AUCs corresponding to Figure

Click here for file

**Table of maximum sensitivities for each specificity.** Rows represent the combination of heritability (H.sq, computed on the observed binary scale) and prevalence (Prev), while columns represent specificities. The elements are the maximal sensitivity in each case.

Click here for file

**Archive containing instructions ****
(readme.txt)
**

Click here for file

Heritability vs. predictive accuracy

**Heritability vs. predictive accuracy.** Relationship of heritability (computed on the observed binary scale) or proportion of variance explained to the maximal upper limit on AUC. The numbers next to the curves represent the prevalence. The maximal AUCs are compared with those that would exist if the genetic risk distribution followed a beta distribution, which is consistent with previous reports

Knowledge of this maximal limit on accuracy is beneficial in the case of type 2 diabetes (T2D), where early targeted intervention can be costly but effective

ROC curves for type 2 diabetes and breast cancer from genomic profiles

**ROC curves for type 2 diabetes and breast cancer from genomic profiles.** Maximal sensitivity / 1-specificity pairs for prediction of type 2 diabetes and breast cancer from full genomic profiles. The maximal pairs are compared to the pairs that would exist if the genetic risk distribution followed a beta distribution, which is consistent with previous reports

Breast cancer has the same maximal AUC as T2D, albeit with a distinct ROC curve from T2D. Breast cancer was found to have a prevalence of 4%

Heritability is the proportion of phenotypic variance explained by

Our method can also be applied in elucidating the maximum accuracy of predictors that integrate features such as gene expression,

Discussion

Our results are general and apply to any binary trait, and they rely on only two commonly estimated parameters. Although the quality of the results is only as good as the estimates of prevalence and heritability for the population in question, our method allows for ranges of prevalences and heritabilities to be considered, which can provide important insight into predictive accuracies. Nonetheless, care must be taken when applying these statistics, as different estimates apply in different situations. For example, in assessing limits to the prediction of lifelong risk, lifelong risk estimates should be used in place of prevalence estimates. In particular, the ballooning lifelong risk of T2D in the USA

The method that we present here can also be used to determine the potential benefit of a future genomewide association study (GWAS) in improving predictive accuracy. To do so, we refer to estimates of GWAS predictive power that were cleverly derived either by simulation studies ^{2} is the broad-sense heritability and ^{2}, is given by

Using this approach, one may evaluate a proposed GWAS based on parameters such as sample size and the number of loci sampled.

Heritability estimates for any binary trait can be used by our method. Broad-sense heritability estimates are needed to cap predictive accuracy, since genetic predictors can exploit dominance and epistatic interactions not measured by narrow-sense heritability estimates. However, if a genetic predictor is constructed as an additive model in line with the assumptions of narrow-sense heritability, then its maximum accuracy can be calculated using narrow-sense heritability; thus, these estimates can also be used, albeit with a slightly different interpretation. Heritability estimates on the normal liability scale can be used after they are transformed to the observed binary scale, e.g. using the method proposed by Dempster and Lerner

Our maximal ROC curves (Figure

Conclusion

We have given exact limits on genetic prediction for any binary trait imposed by the epidemiological parameters of prevalence and heritability. Knowledge of these limits can help delineate the maximal benefits associated with genetic testing, which can allow for cost-benefit analyses, regulation, and clinical guidelines regarding genetic testing even before additional associations are identified. We have also illustrated how these limits can help us prioritize the allocation of research resources, by showing how they can assist in the prioritization and design of future association studies. The calculations presented in this paper could further be used to mitigate the possibility of investing in the development of a genetic test which could never be accurate enough to be of clinical relevance.

Methods

To optimize over the set of risk distributions subject to the disease parameters of average risk and proportion of variance explained (PVE), we modeled a categorical distribution (which resembles a histogram) with _{
i
}, where the _{
i
}‘s (for ^{2} from the regression:

Now, we perform a brief simplification of Equation 1. Following Wray et al. ^{2}:

where _{
i
} is individual

Here, _{
j
} individuals have risk _{
j
} _{
j
}

With this model of the risk distribution and constraints, we can identify the best-case AUC and optimal sensitivity/specificity pairs using the procedures detailed below. Because these procedures associate a single genetic risk distribution with the best-case AUC and a potentially different risk distribution with each optimal sensitivity/specificity pair, it is possible that only some of these sensitivity/specificity pairs may be realizable for a single trait in practice. Consequently, these sensitivity/specificity pairs cannot be used directly to derive the maximal AUC.

Area under ROC curve

To model the AUC, we begin with the random variables

We would like to optimize this term, but unfortunately it is not convex, which would undermine our ability to identify the global optimum. However, after we substitute _{
0
} with

The numerator of this expression can be conveniently represented as ^{
T
}
^{
2
}

We then maximize this AUC over the vector

where the sum of the _{
i
}‘s (for _{
i
} is bounded between 0 and 1.

The parameters

Sensitivity/specificity pairs

To represent each point on the optimal ROC curve, we model the best sensitivity (

Similarly, we can derive specificity:

We optimized sensitivity for any given value of specificity, average risk, proportion of variance explained, and threshold using a linear programming model. This was implemented in the

Calculations for examples

To calculate the proportion of T2D variance explained by physical activity we used Equation 1, where the risk distribution was defined by the prevalence and the relative risks of exercise

Abbreviations

ROC: receiver-operating characteristic; AUC: area under ROC curve; T2D: type 2 diabetes; GWAS: genomewide association study; PVE: proportion of variance explained.

Competing interests

GMC has advisory roles in and research sponsorships from several companies involved in genome sequencing technology and personal genomics (see

Author’s contribution

JMD designed the study, carried out the analysis, and drafted the manuscript. DL designed the study and drafted the manuscript. JEG provided computing resources and helped direct the study. GMC helped direct the study. MFR designed the study and critically revised the manuscript. All authors read and approved the manuscript.

Acknowledgements

We dedicate this to Marco Ramoni, who tragically passed away in June 2010.